Google Blocking Other Search Engine Spiders

from the how-incredibly-not-interesting dept

I'm wondering if this is going to be made into a bigger deal than it really is. Google has put up a robots.txt file (just like many many many other sites out there) telling other search engine spiders to get lost. The article here makes it out as if this is a big deal, which I don't think it is. Google just says they're trying to protect their server resources from being wasted on spiders. Some think that Google is doing it to protect some of their intellectual property from getting harvested. The article gets more interesting when it talks about what other sites have on their robots.txt file (if you don't know, if you put a robots.txt file on your site, you can tell search engine spiders which directories not to look in - of course, this also tells less-than-honest people which directories they probably should look in to find the interesting stuff). Anyway, it turns out that eBay's robots.txt file begins with "Go Away" and CNN's says "Robots, scram".

Leave a Comment..