Daily Telegraph Wants to Sue Google & Yahoo!

I meant to blog about this the other day, but got sidetracked, but apparently The Telegraph wants to sue Google and Yahoo! because they've been crawling it's site.

I'm sorry, but you're taking the piss right?

Danny has a great piece on it here. It's obvious to any webmaster worth his/her salt that you can prevent Google-bot (virtually all varieties) from crawling your site by using the robots.txt command properly and not just put:

Robots.txt file # All robots will spider the domain User-agent: *
Disallow: */ixale/

If you want to exclude the bots from your content learn how to use robots.txt properly and just don't give your lawyer's kids an extra term at private school.

Also if they actually used the Google Webmaster Tools they could see that they could request urls to be removed.

I'm really struggling at the moment to fully understand what's going on with big business and the internet. I thought we'd come to a period now where most companies with bucks in the bank understood the internet - how wrong could I be!

What I'm listening too: