DailyDirt: The Ever-Growing Growth Of Data...

from the urls-we-dig-up dept

There are a lot of reasons to be optimistic about the future. Some folks will always predict doom and gloom, but we say, "The Sky Is Rising!" (loud and proud -- and again with sequel The Sky Is Rising 2). The advent of digital information has created an enormous wealth of data, and the amount of this digital awesomeness seems to be growing all the time. Here are just a few more examples of the amazing abundance of media that surrounds us. If you'd like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post.

Reader Comments

Subscribe: RSS

View by: Time | Thread

  1. identicon
    Rekrul, 24 Jan 2013 @ 11:18pm

    The Internet Archive...

    I don't know if they've changed it, but the Internet Archive used to go too far in obeying the robots.txt files. I once tried to access the archive of a site that no longer existed and was told that the site had been blocked due to a robots file. I was sure this was a mistake as the site was very simple and open while it was up. I was told that whoever owned the domain now, and who had put up one of those parking pages, had probably included a standard robots.txt file.

    In other words; Person puts up site. Internet Archive makes copy of site. Site goes bust. New owner puts up generic site with robots.txt file. IA see robots.txt file and disables access to existing backup of old site.

    When I asked if they couldn't manually override this for sites that are obviously not the same anymore, I was told that it was impossible.

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Show Now: Takedown
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Report this ad  |  Hide Techdirt ads
Recent Stories
Report this ad  |  Hide Techdirt ads


Email This

This feature is only available to registered users. Register or sign in to use it.