Twitter Wishes 4.5 Million Osama Bin Laden-Related Tweets Into Their API Cornfield

from the tweets-or-it-didn't-happen dept

Considering Twitter was instrumental in breaking the story of Osama Bin Laden's death, it seems somewhat strange that they would also be instrumental in limiting access to one of the biggest stories of 2011, if not the decade. (Of course, we're barely into this decade so we probably shouldn't be building these "best of" lists quite yet...) At the center of this unfortunate situation is a dataset constructed from public tweets using either "osama" or "bin laden," which was compiled using Twitter's own API.

Shortly after hearing of Bin Laden's unexpected mortal coil shuffling, Rob Domanski, who blogs as The Nerfherder, was informed of an archive of Osama Bin Laden-related tweets, all packaged up in handy XML format for use with DiscoverText software:

The datafiles were samples taken from live feed Twitter imports starting shortly after the announcement that Osama bin Laden’s death.
  • Twitter searches for "bin laden" (647,585 documents, 505 MB)
  • Twitter searches for "osama" (586,665 documents, 451 MB)
This was all for research purposes, however Twitter quickly shut down the project citing their Terms of Service (TOS) Agreement.

Stuart Shulman of DiscoverText had compiled the documents "using an authorized connection to Twitter via their API" which is apparently a violation of Twitter's API Terms of Service. He received an email from Twitter asking him to remove the datasets:

I'm writing about Twitter data being offered for sale on DiscoverText. Scraping the Twitter service is prohibited by our site Terms of Service, and furthermore, resyndicating data obtained through the Twitter API is prohibited by section I.4.a of our API Terms of Service (

As such, we request you remove the datasets listed at and any other datasets containing Tweets offered on your site.

Shulman responded:

Let’s be clear. We have never sold a Tweet. The data collected through the Twitter API and shared through our system is the same publicly available data other users capture with screenshots and share on blogs, Facebook or Twitter itself. Nonetheless, the datasets we have assembled and similar samples are being taken temporarily off the Web site pending a resolution of this issue with Twitter.

Well, "temporarily" has turned into "indefinitely." As of June 1st, Shulman's dataset contained 4.5 million Osama Bin Laden-related tweets, all of which can only be marveled at as a REALLY BIG NUMBER but not shared in any usable fashion thanks to Twitter's complaint.

If it's just a "policy first" decision on Twitter's part, it seems a little short-sighted. This information was (and is) of great interest to people worldwide. Perhaps some sort of warning could have been issued instead of a full takedown, thus allowing Twitter to assert its position on API usage without locking up the dataset. Once the dataset already exists, why block it? It's disheartening to see something with as much potential as Shulman's project getting thrown under the TOS bus.

Reader Comments (rss)

(Flattened / Threaded)

  1. identicon
    Anonymous Coward, Jun 17th, 2011 @ 4:20pm


    I thought only the evil 'MAFIAA' cared about protecting their content in a digital age? And all these upcoming tech companies were about 'openness' and a new way of doing things?

    You people are ridiculous.

    reply to this | link to this | view in thread ]

  2. identicon
    Anonymous Coward, Jun 17th, 2011 @ 4:21pm

    This information was (and is) of great interest to people worldwide.

    I lol'ed. At least you didn't say great importance.

    reply to this | link to this | view in thread ]

  3. identicon
    Anonymous Howard, Cowering, Jun 17th, 2011 @ 4:55pm


    I wonder if the FBI or some other governmental/cryptically-initialled agency leaned on Twitter...

    Time to adjust the tin-foil hat. They can hear you, you know.

    reply to this | link to this | view in thread ]

  4. identicon
    Anonymous Coward, Jun 17th, 2011 @ 5:18pm

    List of Idiots

    This is a fine example of Party A objecting to Party B making A's data more valuable, for free. Mike has taught us well to recognize the pattern. Then A engages in some sort of sleaze to make the data less valuable again. Isn't that the kind of management boneheadedness which is supposed to be restricted to members of the XXAA and the government? Looks like Twitter wants to be added to the list of idiots.

    reply to this | link to this | view in thread ]

  5. identicon
    Rekrul, Jun 17th, 2011 @ 5:24pm

    "It's a good thing you done there Twitter! A real good thing!" :)

    reply to this | link to this | view in thread ]

  6. identicon
    Anonymous Coward, Jun 17th, 2011 @ 5:24pm

    How come it wasn't just upped as an xml file over bittorrent?

    reply to this | link to this | view in thread ]

  7. icon
    Miff (profile), Jun 17th, 2011 @ 5:52pm

    I'm honestly surprised Twitter still has an API capable of doing anything useful.

    reply to this | link to this | view in thread ]

  8. identicon
    Anonymous Coward, Jun 17th, 2011 @ 7:44pm

    War on Terror is a hoax...

    reply to this | link to this | view in thread ]

  9. identicon
    Androgynous Cowherd, Jun 18th, 2011 @ 1:51pm

    I'd like to know under what legal theory Twitter has any right to demand that data taken down. Surely the copyrights in the individual tweets are vested in the individual tweeters, and the United States doesn't recognize a separate database right in a mere compilation of separately copyrighted works where that compilation doesn't add any original creative expression (e.g. in the arrangement) -- likely these archives are just chronological, or even disordered.

    So I don't think Twitter has a copyright claim. And I don't see any other IPR being remotely applicable here. Since the tweets were public, trade secrecy clearly cannot apply, and they're obviously not patentable, nor can Twitter have trademarked them, though the tweets may contain trademarks here and there. If there's a publicity rights violation in there anywhere, again the right being violated would be an individual tweeter's and not Twitter's. (That's leaving aside the question of whether a 140-character tweet contains enough creative expression to even be copyrightable at all.)

    Which means that Twitter hasn't a legal leg to stand on if that site operator puts the archives back up and keeps them that way. The most Twitter can do about an alleged TOS violation is a) terminate the alleged violator's Twitter account and possibly b) sue for breach of contract. But they have no proprietary interest in that data, legally speaking.

    reply to this | link to this | view in thread ]

  10. icon
    Tom Landry (profile), Jun 18th, 2011 @ 1:58pm

    Utterly fantastic title for this piece Tim....lolol

    reply to this | link to this | view in thread ]

  11. icon
    Josh in CharlotteNC (profile), Jun 20th, 2011 @ 6:13am


    I'd like to know under what legal theory Twitter has any right to demand that data taken down.

    The data was gathered using Twitter's API.
    In order to use the API, you must agree to a TOS.
    TOS and EULAs are considered contracts, even though no one reads them.

    I'm not a lawyer, but as far as I remember, no one's really been willing to decisively challenge or defend click-through and shrink-wrap TOS and EULAs for fear of a judge making a ruling that turns out to be a precedent.

    reply to this | link to this | view in thread ]

Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
Insider Shop - Show Your Support!

Hide this ad »
Essential Reading
Techdirt Deals
Techdirt Insider Chat
Hide this ad »
Recent Stories
Advertisement - Amazon Prime Music
Hide this ad »


Email This

This feature is only available to registered users. Register or sign in to use it.