News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful
from the deep-misunderstandings dept
Following on the speech given earlier this month by the head of the Associated Press, where it was made clear that the AP and news organizations still think that they can be gatekeepers of news, a bunch of publishers along with the AP are now trying to revise robots.txt so that they can hide content on a more selective level. Now, it is true that robots.txt can be rather broad in its sweep. But it’s rather telling that it’s the publishers who banded together and are telling search engines what changes are needed, rather than working with the search engines to come up with a reasonable solution. In the meantime, there really are some simple solutions if you don’t want content indexed by search engines — but we’ve yet to fully understand why publishers are so upset that Google, Yahoo and others are sending them so much traffic in the first place.
Filed Under: publishers, robots.txt
Companies: associated press
Comments on “News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful”
Simple solutions:
A: Put the content you don’t want crawled on a different server and blacklist all known bots from connecting to that server.
B: Use a login script.
I've said it before...
and I’ll say it again – Google, Yahoo, and the others need to completely remove news publishers that complain from their listings. Poof! Gone! Problem solved!
Unfortunately for the publishers no more listings equals no more traffic. Bye-bye AP, bye-bye archaic newspapers that can’t make the move into the 21st century.
If the search engines would do this the news publishers would quickly see the value of the listings.
More info on proposed changes ?
Wish there was more information on the proposed changes. The Acap website is also not very clear. There is some information on their reasoning in their FAQs.
update:
A pdf document with the proposed changes.
Why it does make sense
The fact that there are no business models that make use of this fine-grained control now doesn’t mean there won’t be such in the future. They are smartly making sure the technology is already in place so they are free to experiment with such business models, and if they find a successful one stick with it. While it’s possible they won’t find a successful model, they certainly won’t find it without the technology to back it up.
You may believe that no such model exists, but bear in mind that a failed experiment is as important as a successful one. For example, if the music industry started off selling unprotected MP3, they would probably get scared of the inevitable piracy, however minor, that would ensue and fold out of the digital media market. Now that they tried various DRM schemes – and failed miserably – they are much more committed to selling unprotected digital media and much more pragmatic as to what is an acceptable level of piracy. Similarly, while such an experimentation period may not help the publishing industry actually find a better Internet strategy, it will certainly make sure they are more informed and committed when they do pick their strategy.
And anyway, having more control can never be bad. Since the courts seem to agree the publishers do have a say as to how their content is used, they might as well use that power to put in place the technology to enforce control. Then they can decide to just continue with the status quo, and change it when they find it beneficial.
Be patient. ACAP is a proposal and voluntary. Those that that would need to use it will decide if it has value.