News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful

from the deep-misunderstandings dept

Following on the speech given earlier this month by the head of the Associated Press, where it was made clear that the AP and news organizations still think that they can be gatekeepers of news, a bunch of publishers along with the AP are now trying to revise robots.txt so that they can hide content on a more selective level. Now, it is true that robots.txt can be rather broad in its sweep. But it’s rather telling that it’s the publishers who banded together and are telling search engines what changes are needed, rather than working with the search engines to come up with a reasonable solution. In the meantime, there really are some simple solutions if you don’t want content indexed by search engines — but we’ve yet to fully understand why publishers are so upset that Google, Yahoo and others are sending them so much traffic in the first place.

Filed Under: ,
Companies: associated press

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful”

Subscribe: RSS Leave a comment
Johan says:

I've said it before...

and I’ll say it again – Google, Yahoo, and the others need to completely remove news publishers that complain from their listings. Poof! Gone! Problem solved!

Unfortunately for the publishers no more listings equals no more traffic. Bye-bye AP, bye-bye archaic newspapers that can’t make the move into the 21st century.

If the search engines would do this the news publishers would quickly see the value of the listings.

Carme says:

Why it does make sense

The fact that there are no business models that make use of this fine-grained control now doesn’t mean there won’t be such in the future. They are smartly making sure the technology is already in place so they are free to experiment with such business models, and if they find a successful one stick with it. While it’s possible they won’t find a successful model, they certainly won’t find it without the technology to back it up.

You may believe that no such model exists, but bear in mind that a failed experiment is as important as a successful one. For example, if the music industry started off selling unprotected MP3, they would probably get scared of the inevitable piracy, however minor, that would ensue and fold out of the digital media market. Now that they tried various DRM schemes – and failed miserably – they are much more committed to selling unprotected digital media and much more pragmatic as to what is an acceptable level of piracy. Similarly, while such an experimentation period may not help the publishing industry actually find a better Internet strategy, it will certainly make sure they are more informed and committed when they do pick their strategy.

And anyway, having more control can never be bad. Since the courts seem to agree the publishers do have a say as to how their content is used, they might as well use that power to put in place the technology to enforce control. Then they can decide to just continue with the status quo, and change it when they find it beneficial.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...