Hilton.com Robots.txt Gaff

You've gotta love some people. You get some people in this industry pretending to know what they're talking about but they just end up showing themselves as numpties.

I don't know who is responsible for the robots.txt file on the Hilton.com site but they sure need a slap!

Take a look at the first two lines:

# Daytime instructions for search engines
# Do not visit Hilton.com during the day!

Gawd, did I chuckle when I read this!

But it does raise an interesting point: Wouldn't it be great if you can say to bots "don't crawl between 9am and 5pm"? But obviously it'll be 9am whose time? There has been a suggestion that this can be done in the "Extended Standard for Robot Exclusion", however. My only concern is that if it's based on something like (GMT) which it realistically has to, we would regularly have to change the file to account for daylight savings which obviously doesn't shift at the same time each year and the same time accross the world.

The simple solution, however, is to use "Crawl-delay: 10" (that's seconds) so that if you do have a huge site and a crappy host you can make sure that the site isn't massively effected.

However, Google does actually throttle it's crawl rate when it considers it's putting too much pressure on your server.

For those that are still feeling concerned about their crawl rate, consider adding this to your robots.txt:

User-agent: msnbot
Crawl-delay: 10

User-agent: Teoma
Crawl-delay: 10

User-agent: Slurp
Crawl-delay: 10

For more information about the robots.txt file and seo try the search on the top right-hand side of the page, or look at Wikipedia.