Twitter Wishes 4.5 Million Osama Bin Laden-Related Tweets Into Their API Cornfield

from the tweets-or-it-didn't-happen dept

Considering Twitter was instrumental in breaking the story of Osama Bin Laden’s death, it seems somewhat strange that they would also be instrumental in limiting access to one of the biggest stories of 2011, if not the decade. (Of course, we’re barely into this decade so we probably shouldn’t be building these "best of" lists quite yet…) At the center of this unfortunate situation is a dataset constructed from public tweets using either "osama" or "bin laden," which was compiled using Twitter’s own API.

Shortly after hearing of Bin Laden’s unexpected mortal coil shuffling, Rob Domanski, who blogs as The Nerfherder, was informed of an archive of Osama Bin Laden-related tweets, all packaged up in handy XML format for use with DiscoverText software:

The datafiles were samples taken from live feed Twitter imports starting shortly after the announcement that Osama bin Laden’s death.

  • Twitter searches for "bin laden" (647,585 documents, 505 MB)
  • Twitter searches for "osama" (586,665 documents, 451 MB)

This was all for research purposes, however Twitter quickly shut down the project citing their Terms of Service (TOS) Agreement.

Stuart Shulman of DiscoverText had compiled the documents "using an authorized connection to Twitter via their API" which is apparently a violation of Twitter’s API Terms of Service. He received an email from Twitter asking him to remove the datasets:

I’m writing about Twitter data being offered for sale on DiscoverText. Scraping the Twitter service is prohibited by our site Terms of Service, and furthermore, resyndicating data obtained through the Twitter API is prohibited by section I.4.a of our API Terms of Service (

As such, we request you remove the datasets listed at and any other datasets containing Tweets offered on your site.

Shulman responded:

Let’s be clear. We have never sold a Tweet. The data collected through the Twitter API and shared through our system is the same publicly available data other users capture with screenshots and share on blogs, Facebook or Twitter itself. Nonetheless, the datasets we have assembled and similar samples are being taken temporarily off the Web site pending a resolution of this issue with Twitter.

Well, "temporarily" has turned into "indefinitely." As of June 1st, Shulman’s dataset contained 4.5 million Osama Bin Laden-related tweets, all of which can only be marveled at as a REALLY BIG NUMBER but not shared in any usable fashion thanks to Twitter’s complaint.

If it’s just a "policy first" decision on Twitter’s part, it seems a little short-sighted. This information was (and is) of great interest to people worldwide. Perhaps some sort of warning could have been issued instead of a full takedown, thus allowing Twitter to assert its position on API usage without locking up the dataset. Once the dataset already exists, why block it? It’s disheartening to see something with as much potential as Shulman’s project getting thrown under the TOS bus.

Filed Under: , ,
Companies: twitter

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Twitter Wishes 4.5 Million Osama Bin Laden-Related Tweets Into Their API Cornfield”

Subscribe: RSS Leave a comment
Anonymous Coward says:

List of Idiots

This is a fine example of Party A objecting to Party B making A’s data more valuable, for free. Mike has taught us well to recognize the pattern. Then A engages in some sort of sleaze to make the data less valuable again. Isn’t that the kind of management boneheadedness which is supposed to be restricted to members of the XXAA and the government? Looks like Twitter wants to be added to the list of idiots.

Androgynous Cowherd says:

I’d like to know under what legal theory Twitter has any right to demand that data taken down. Surely the copyrights in the individual tweets are vested in the individual tweeters, and the United States doesn’t recognize a separate database right in a mere compilation of separately copyrighted works where that compilation doesn’t add any original creative expression (e.g. in the arrangement) — likely these archives are just chronological, or even disordered.

So I don’t think Twitter has a copyright claim. And I don’t see any other IPR being remotely applicable here. Since the tweets were public, trade secrecy clearly cannot apply, and they’re obviously not patentable, nor can Twitter have trademarked them, though the tweets may contain trademarks here and there. If there’s a publicity rights violation in there anywhere, again the right being violated would be an individual tweeter’s and not Twitter’s. (That’s leaving aside the question of whether a 140-character tweet contains enough creative expression to even be copyrightable at all.)

Which means that Twitter hasn’t a legal leg to stand on if that site operator puts the archives back up and keeps them that way. The most Twitter can do about an alleged TOS violation is a) terminate the alleged violator’s Twitter account and possibly b) sue for breach of contract. But they have no proprietary interest in that data, legally speaking.

Josh in CharlotteNC (profile) says:

Re: Re:

I’d like to know under what legal theory Twitter has any right to demand that data taken down.

The data was gathered using Twitter’s API.
In order to use the API, you must agree to a TOS.
TOS and EULAs are considered contracts, even though no one reads them.

I’m not a lawyer, but as far as I remember, no one’s really been willing to decisively challenge or defend click-through and shrink-wrap TOS and EULAs for fear of a judge making a ruling that turns out to be a precedent.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...