Cross-posted from ninapaley.com

Dear Internet,

You know what should be really easy to find online? Good quality, Public Domain vintage illustrations. You know, things like this:



I found this on Flickr, where someone claims full copyright on it. That's copyfraud, but understandable because Flickr's default license is full copyright (all the more reason to ignore copyright notices!). But copyfraud isn't not the main problem. The main problem is that images like this are painfully difficult to find online, especially at high resolutions (and this image is only available at medium resolution - up to 604 pixels high, which is barely usable for most purposes but higher than much of what you find online).

The images are out there - and with zillions of antique books being scanned, their vintage illustrations are being scanned right along with them. But the images are buried in the text, and often the scan quality is poor. Images should be scanned at high quality, and tagged for searchability.

Are archives ignoring the value of images?

Take the American Memory archive of the Library of Congress. Lots and lots of historical documents here, but no way for me to find an image of, say, a horse.

Most book-scanning projects focus on texts, not illustrations. Many interesting and useful illustrations are buried within these scans, uncatalogued and inaccessible. Scan quality is set for text, not illustrations, so even if one can find a choice illustration buried within, its quality is usually too low to use.

Archive.org is great (I love you, archive.org!) but does not have an image archive. Still images are not among their "Media Types" (which consist of Moving Images, Texts, Audio, Software, and Education). So I went spelunking through their texts, starting with "American Libraries," and searched for something easy: "horse." Surely I could find a nice usable etching of a horse in there somewhere. I eventually found "The Harness Horse" by Sir Walter Gilbey, from 1898.



Nice illustrations! Can I use them? Unfortunately, no. The book is downloadable as PDF and various e-publication formats, but when I try to extract the illustrations, I get a mess (which you can see, after the jump):

Having recently been part of an effort to preserve videos on Google Video, in light of its announced shutdown, it's made me more aware how brittle our current culture is, with many many artifacts available only in digital form. There are conservation efforts such as the aforementioned one, part of a larger group called Archive Team. Google changed its mind and promised to keep Google Video online and try to move as much as possible to YouTube where it has permission from the original uploader. This is, obviously, a win for the archivists, who kept the effort going just the same and are still in the process of uploading all of Google Video to archive.org.

Contrast this with Friendster, where the Archive Team project was unable to save everything before the end of May, when everything was set to be deleted. Much has been downloaded and it will still give a good picture of what this early social network was like. Similarly, the Archive Team was able to rescue much, but not all of GeoCities, before Yahoo shut that down last year, releasing its collection as a massive 1 TB bittorrent file.

Now you may ask yourself is Friendster (or even GeoCities) worth saving? Answering that question purely myself, I'd say no, but I have no connection with those sites. Looking at the question however from the perspective of people who have spent many hours building these profiles and interacting with each other, I can see there's clearly value there to those who used them. Answering instead as someone with a deep interest in history, it's not for us to say what will eventually prove worth saving. Instead, that's something for the historians, archeologists, sociologists and other interested parties in the future to decide. But they can't do that if the information isn't even there.

How often have we not wished more of a particular point in our history had been preserved? Ironically many of the analog writings of our past are in a better state than some of our digital ones, even when it's often pointed out how easy it is to make bit-perfect copies of something in this day and age.

All of this, leads me to ask the following questions:

• Do you agree the digital aspects of our culture should be better preserved?
  
• Should we have, maybe even one on each continent or in each country, a modern Library of Alexandria? (identical copies in different places to prevent 'a fire' from destroying it)
  
• What would such a world-wide archival effort look like, technologically?
  
• What should we do about copyrights and patents (and DRM) that might get in the way of preserving our heritage?
  
• How do we go about archiving software for a computer that's available only in a museum, to preserve interactive access?
  
• What about the proliferation of file formats? Do we transform everything into a canonical form where we can, or do we store the original along with software to interpret?
  

I think we owe it to ourselves that this generation doesn't look like a black hole when viewed from the future, and by extension, I'd like to make it easier on future historians to learn about us than it has been for us to learn about previous societies (e.g. Sumerians).

I'll start off by saying that though little personal correspondence has been preserved of earlier times, we should recognize the right to privacy and any such posited archive would be allowed to include private communications only if you explicitly opted in. Valuable though these are for historians, if we preserve as much as we can of the digital domain, enough can be inferred from context that this would be an unwarranted intrusion.

But what about the wider issue of preservation of public content that can be wiped out by shutting off a power switch?

Last month, the BBC announced its intention to kill off a bunch of websites, including Douglas Adams' old stomping grounds, H2G2. Apparently some of the sites were going to be archived, and others weren't -- but as you may know, the BBC does not have the greatest of reputations when it comes to archiving old material. However, this is the internet. If someone announces they are going to get rid of something that other people would like saved, there are tools to save it. Glyn Moody points us to a note from Ben Goldacre about how some "anonymous nerd" archived all of the sites the BBC is set to take down (it cost him a whopping $3.99) and he's now set that archive free, so you, too, can help make sure this content lives on. The whole thing is available via a torrent file at http://178.63.252.42/, which also has a description of the project, so have at it.

Recently, we pointed out that various film archives were disintegrating, and noting that perverse copyright laws were partly to blame. Now, Copycense points us to the news that experts are also quite worried about audio recordings degrading and disappearing -- including recent recordings, such as from 9/11 and the 2008 election. This is part of a study commissioned by the Library of Congress -- so it's not just some random researchers. Thankfully (unlike the article on films) the study and the reporting about it highlights the copyright issue:

A hodgepodge of 20th century state anti-piracy laws also has kept most sound files out of the public domain before U.S. copyright law was extended to sound recordings in 1972. The study found only 14 percent of commercially released recordings are available from rights holders. That limits how much preservation can be accomplished, Brylawski said.

The full study (The State of Recorded Sound Preservation in the United States: A National Legacy at Risk in the Digital Age (pdf)) suggests that copyright laws be changed to deal with the archival problem. The study mentions the famous (and mostly ignored) UK Gowers study that found that extending copyright was a mistake (Gowers later admitted, separate from the report, that the economic evidence clearly indicated copyright terms should be shortened). The study also quotes many researchers, academics and archivists who point out how copyright law has made it quite difficult for them to do basic research and other useful things without violating the law.

While fair use is frequently invoked, there is little case law interpreting its application to specific situations. Some archivists believe the lack of applicable case law discourages institutional general counsels from authorizing preservation and access programs on the basis of fair use. Fair-use provisions in the copyright law (Section 107) permit use of excerpts of copyrighted materials, including recordings, for educational and scholarly purposes, for citation in news reporting and criticism, and for nonprofit activities. However, the courts have interpreted fair use to be an affirmative defense. In other words, the burden is on the defendant to prove fair use; the plaintiff does not need to establish that the use falls outside fair use. Besek further notes that "favored uses are not automatically deemed fair, and other uses are not automatically unfair. The determination depends on the facts of a particular case."

Most witnesses at the hearings conducted in support of this study who were affiliated with educational institutions expressed a belief that copyright law as it applies to sound recordings is too complex to interpret easily, too restrictive, or both. Section 108, for example, which limits access to digital copies made available under that section to library premises, is too narrow to address all educational needs. One witness called for "premises" to be expanded to include network domain, in order to accommodate "the manner in which students and scholars use information in the current academic and scholarly environment. Increasingly, learning is occurring off-site, that is, at home, in the dorm or just anywhere on campus, in addition to the classroom and library."

Separately, the report notes that DRM and the DMCA's anti-circumvention provision is a real problem for researchers and archivists.

In the end, the study says that copyright law needs to change to deal with this very real challenge:

Creation of new copyright laws or licensing procedures that acknowledge best practices in audio preservation and assure access to audio heritage is essential to ensure the preservation of that heritage and its understanding and appreciation by generations to come.

It then repeats five suggestions made by The Association for Recorded Sound Collections (ARSC), such as changing the fact that pre-1972 recordings are all limited under ancient state laws, rather than federal copyright law (meaning they won't be in the public domain for much, much, much longer than more modern recordings). They also suggest moving copyright term back (just slightly) to 50 or 75 years, supporting orphan works legislation, enabling compulsory licensing to copyright holders for anyone wishing to reissue "abandoned" recordings and allowing non-profits to backup and archive sound recordings without violating copyright. The recommendations are, frankly, a bit weak, but in an era where the government only seems to want to strengthen copyright, it's nice to see at least some effort to push back in some areas. Of course, the likelihood of the Library of Congress actually following through on any of these recommendations seems slim at best.

While UK politicians are arguing over a ridiculous and unnecessary change to copyright law as part of the Digital Economy Bill, it appears there's a much bigger problem with UK copyright law that isn't getting very much attention at all. Slashdot points us to the news that due to the way copyright law currently works in the UK, archiving websites without permission is illegal. Yes, even for the British Library and other institutions who are designated by law to keep a copy of every printed publication. But when it comes to the web, the Library needs to get permission from every website that it wishes to archive. Obviously, that greatly limits the archival activity that the Library can be involved in -- and, as a result, the public suffers greatly. This is a clear case where fair use should cover the issues, but current law does not adequately handle this. Making fair use work better should be a priority -- but instead we have politicians trying to prop up Hollywood's business model by pushing copyright law in the other direction.

A bunch of folks have sent in the news that the Economist appears to be putting up something of a paywall, locking up all archival content older than 90 days, while also locking up one version of the magazine (the one that is made to look just like the physical paper layout). I have to be honest: I don't see how this makes any sense at all. In our experience, somewhere between 25% to 30% of our daily traffic is to archival content, usually in the form of search engine traffic -- or occasionally other sites picking up on an older story. Archival content is perfect Google fodder, driving traffic (and ad views) to pages that otherwise would get no traffic at all. In many ways, that's a big part of the value of having widespread archives -- to bring in such traffic for those who care about it. The chances of such a "drive by" viewer paying up for a subscription to view that content seems incredibly slim -- and it seems quite likely that the decline in traffic (and ad dollars) would almost certainly outweigh the number of new subscribers added. This doesn't seem to make any sense at all. Does The Economist have any information economists on staff?

Back at the end of March, we were surprised that the NY Times, in consolidating its regular site with the site of the International Herald Tribune (which it owned) had broken all the links to IHT.com. Rather than taking them to the article in question on the NY Times site, it simply took them to a landing page. This was just a bad idea all around. It appears that a former reporter for IHT, Thomas Crampton, discovered this over the weekend and has brought renewed attention to the issue by issuing an open letter to the NY Times asking why it "deleted" his career -- in that all of his early work that appeared in the IHT is now gone (some, but not all, of it remains in the NY Times). Additionally, he pointed out that this is also causing problems for Wikipedia, notably with any article that relied on evidence from an IHT article. While we've seen others erase old articles as well (and the Associated Press is famous for forcing all its partners to take down AP articles after just a short time period), it still is amazing in this day and age that anyone thinks it's a good idea to break links to news stories -- especially when the value of archives found via search engines is so high.

Plenty of folks have pointed this out for years, but newspapers that try to lock up their back archives and charge for viewing those articles are very likely hurting their bottom line more than helping it. That's because those archives are a treasure trove of info that people would be interested in finding via a search engine -- but they almost never want to pay for it. For many years, the NY Times tried locking up its archives and charging to read stories, but eventually did the math and realized it made a lot more sense to put all its archives online for free, and make money off the ads. Since removing the barriers, the NY Times has seen its traffic spike significantly, and its archives have become a significant portion of the overall site's traffic.

However, some newspapers still can't see the forest for the trees, and think that the answer is to charge high prices to view old articles. That most likely just gets people to look elsewhere, and diminishes ad revenue as well. Parker Mason has written an open letter to the Toronto Globe & Mail decrying its continued practice of charging $5 for access to a single archived article (for just 30 days of access). It's a good read, and I'm guessing that folks like Mathew Ingram, who works at The Globe, have been pushing for changes to the paper's policy, but until then, the company seems to be hurting itself. Mason's letter is well worth reading, but here's a snippet:

But then you go and do something like trying to charge me $4.95 for a newspaper article that I've already paid for and read, and this hurts me (telling me that this content will only be available for 30 days only adds insult to injury).

Your greatest asset is the thousands and thousands of pages of information and news stories that you have in your archives. People want to view this content, and just as they have endured advertising in your print publications, they'll endure the same kind of advertising on your website.

I understand your thinking when it comes to locking up this content behind a pay wall: it is valuable information, so people will pay to see it.

The problem is, you are only half-right. It is valuable information, but only when it is easy to access. In the age of Google, people will quickly move on and find the information elsewhere, somewhere where it easier to get at.

Copyright was clearly designed for a different age: when not everyone was a "publisher." And while we've spent years pointing out many of the different problems that has caused, here's another one: how is a library or some other institution charged with "archiving" written works for posterity supposed to deal with copyright laws that can often make such archival activities against the law? Well, the Library of Congress and a bunch of other organizations have a suggestion: let them all ignore copyright law for the sake of archiving. Basically, the report recommends that certain organizations be designated as "preservation institutions," which are then more or less allowed to ignore copyright law and copy-at-will for the sake of preservation. Of course, this is clearly going to lead to many questions, including just who would get designated as such. Many people can probably agree on public libraries and such -- but what about Google? After all, Google is already one of the largest players in "preserving" what's online and also, with its book scanning project, what's in books. Yet it's a private, for-profit company. Should it qualify? I would argue that it makes sense to allow it, given how beneficial the archival activities of Google have already been. Even if it is for profit, the public benefit has been tremendous as well. But then what's to stop any other company from arguing that it to deserves an exemption for preservation purposes? Wouldn't a better solution be to start rethinking copyright law altogether, since what has become clear from this is that copyright doesn't quite fit today's world any more?

In the wake of the ridiculous dustup between bloggers and the AP, where the AP threatened bloggers who help promote AP articles, some are starting to point out that the AP's problem goes well beyond a seriously distorted view of copyright law. The reason this came up at all was because the AP's business model is pretty screwed up in a web world. This was clear from the fact that the more involved the AP gets online the more it ends up competing with all its member newspapers. Almost every action it takes seems to help the AP's business model, while hurting its members. With the latest skirmish, the AP is ensuring that those partner sites get fewer links in and less traffic. Back when the AP signed a deal with Google News, we pointed out that its member newspapers should be pissed off. Basically, the AP and Google had worked out a deal to keep traffic away from the member papers. That's no way to survive. Eventually, if this keeps up, those members freak out and stop supporting the AP.

That leads to great suggestions from Danny Sullivan on how it's time for the AP to totally rethink its business model, recognizing what the web does for its business:

Well, wake up call. You need a new model. Really. Or you're going to die.

The AP should have a news portal. You should take in content from your members, put it up in an easy-to-find way and generate the ad dollars to be redistributed back to your members. Like do it now, before since the entire licensing thing ain't going to live that long.

It's worth reading the whole thing, as it's right on point. However, there is one additional thing that's worth mentioning: the AP really needs to learn to keep its news online. Back when I would link to AP articles, one of the most annoying things was that they would disappear after a couple weeks. We used to get complaints all the time from readers who would find an older post and the underlying AP article would be gone. I had thought that maybe its deal with Google would change this, and started pointing to the Google versions of AP articles... but, nope, those disappeared after a few weeks as well.

In the narrowminded world of an AP exec, they probably think this leads to more licensing revenue, since it will make people search out and license the article after it can't be found any more. Nope. It just makes people pissed off. Many newspapers have realized that there's tremendous value found in freeing up the archives, and monetizing that long tail of traffic via advertisements. That's a lot more effective than pissing off large groups of folks (including the people who promote your articles) and hoping it leads to a little more licensing revenue. So, while I agree with Danny's recommendations for the AP to join the internet era, I'd also add a recommendation that it open up its archives and recognize that URLs should be permanent rather than fleeting.