Database CIA Claimed Too Difficult To Compile For FOIA Requesters Released In Full On CIA Website

from the three-years-of-fuckery dept

The CIA has millions of declassified records stashed away in Maryland — something it claimed was accessible to the public. Actual access, however, resembles something out of Terry Gilliam’s “Brazil,” rather than what any reasonable person would call “accessible.”

This so-called “publicly accessible” database — known as CREST — has been the target of MuckRock contributor Mike Best, who kickstarted an effort to liberate records from the vault through the use of manual labor. The records can be accessed by computer, but only certain computers, and only if you know exactly where to find them.

This is the CIA’s “publicly available” records system front-end.

And here’s how you locate it.

Accessing the information isn’t easy. Researchers have to go to the back of the 3rd floor library at the National Archives building in Maryland, which is unfortunately unstaffed for half the day. Tucked away in the library are the only computers that can access the millions of pages of declassified records. If researchers ask the the main “Information” desk, they’re answered with confused stares and incorrect directions. Researchers trying to look up on the National Archive’s website where to access the computers, won’t find it on the page about doing research at that location or on the page for electronic records at that location. That information is tucked away on the page for online databases – despite not being online.

Best’s crowdfunding effort sought to free these documents from their four-computer cage. But the only way to do so was to visit in person, print out pages, and rescan them. All of this would be done under multiple forms of surveillance at the National Archive.

To expedite the process, MuckRock turned to litigation. Three years after commencing its FOIA lawsuit against the CIA, MuckRock has emerged victorious. The CIA has released the contents of the CREST vault online at its site, something it repeatedly claimed would take dozens of years and hundreds of thousands of dollars to complete.

The story behind the CIA’s forced transparency is amazing. The claims made by the CIA during its opacity efforts are simply astounding. MuckRock has published a long, detailed recounting of its FOIA battle against the agency at its site and I wholeheartedly encourage you to click through and read it. But here are some of the highlights.

First off, everything in the database is in an unsearchable format by choice. The CIA only uses TIFF files, claiming that these are more resistant to alteration. But when faced with litigation, the CIA reversed course on the supposed hardiness of TIFF files.

The declaration… says that CIA cannot release these TIFF files in electronic form because they can be so easily altered by the mere act of a CIA FOIA analyst looking at them, and that the security measures they must take to remove this accidental metadata for an electronic release (involving editing each file separately by hand) would take 28 years and 1,200 CDs.

And there’s where the hilarity begins. These are the DOJ’s claims as to the difficulty of releasing the database it just released well ahead of its 28-year estimate. Not only is the process supposedly far too onerous to even begin to consider undertaking, but the DOJ claims the documents it just made available to the public at the CIA’s website are not of public interest — this despite being (technically) available to for public viewing at the National Archive for several years now.

In another filing, the CIA admits the 1,200 CDs that would take 28 years to compile have actually already been compiled — and there are actually 1,450 CDs of records, but it will only consider releasing the quoted 1,200 if forced to. Unless the CIA is still porting these documents over to the CREST system at the National Archive, its claim of “28 years” looks even more ridiculous. From one of MuckRock’s filings:

“Moreover, when [CIA] estimates that it would take 28 years just to create copies of the CREST database, it begs the question of how CIA loaded CREST in the first place. The system has only been operational since 2000.”

Then it claimed it would need $108,000 and six months to make copies of the CDs it already had in hand. And it doubled down on its contradictory claims about TIFF files, stating they were so easy to alter because they were so difficult to alter.

This lead to one of the best rebuttals (and lead-offs) in FOIA history:

I file a sur-reply drawing on my extensive experience as a person with a working brain, closing with the following thought:

“Last, CIA claims that ‘the act of a CIA employee opening a document on his or her terminal may cause metadata to embed itself on the image header.’ This is a frivolous statement for two reasons. First, files on PCs are not altered unless they are saved after the alteration. Simply opening a file and then closing it without changing it does not embed metadata on the file. Second, if this were a valid concern, it would apply to every file processed by the CIA FOIA office, not just CREST files. Since it clearly does not (which would paralyze the FOIA office), then the Court should view this claim very skeptically.”

Kel McClanahan — the author of the post and MuckRock’s legal rep for this case — digs into the DOJ/CIA’s repeated assertions about the “burdensomeness” of complying with this FOIA request. By the time he’s done, everyone and everything is covered with government bullshit.

In 2000, using cutting edge 2000 technology, CIA populates CREST over a matter of months. For the next 15 years they insist that the only way to protect this system is to videotape people who want to access it. In 2015 they say that, using cutting edge 2015 technology, it will take them 28 years to make CD copies of CREST. Then they say that they already MADE CD copies of CREST in a matter of months if not weeks, meaning that it will only take 6 years to copy THOSE copies.

But they can copy those records to the web in 4 years. Less than a year later, they say that because of the huge public interest in CREST over the last several years (that they expressly disavowed less than 6 months before, did you remember?) they will have all of CREST online within a year because nobody needs to be videotaped any more. Then, less than 2 months later, they put all of CREST online.

The CIA and DOJ insist this stonewalling had nothing to do with the CIA’s online release of CREST files. Apparently, it did all of this out of the goodness of its heart, forfeiting a shot at $100,000+ in FOIA fees and chance to do next to nothing for the next 30 years. It’s obvious the effort to move these records online increased once it became apparent the government was going to come out on the losing end of these FOIA lawsuits (Jason Smathers was suing the CIA over the same database). Rather than be on the hook for legal fees, the CIA preempted any judicial judgment by releasing them to the public (the same public it said had “no interest” in these files) in bulk before final rulings could be made.

Filed Under: , ,
Companies: muckrock

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Database CIA Claimed Too Difficult To Compile For FOIA Requesters Released In Full On CIA Website”

Subscribe: RSS Leave a comment
Anonymous Coward says:

Re: Re:

The problem is that a “publically available” database is relatively easy to cover for media which eventually would force some politicians into claiming terrible things about their handling of data.

Releasing data after it is clear how bad of a court case they have is, stalling to extort the victim and buy time to prepare a way to make the case disappear and then claiming it was all part of a plan is pretty much standard government practice.
Losing in court looks extremely bad and would create even worse precedence, forcing them to comply with the letter of the law instead of having a goto excuse for opacity. Preemptively complying to make the court case go away will be possible to spin into something harmless in the media since very few journalists bring anything from court documents (the journalist would be crucified by their overlords because of how boring the story will be.).

Anonymous Coward says:

Re: You're not understanding the CIA's problem

One problem is wmds (chemical/biological). Another problem is incapacitation due to mind control(particle beam weapons). Then there is cia torture.

Stop torturing me. Stop torturing the cia. Stop bragging that you torture me and the cia enough to have been al qaeda designated for so many decades.

Eldakka (profile) says:

Re: CDs?

Wonder how long they’d claim for machine-readable punch cards?

One 80-column punch card can hold about 72-bytes plus an 8-byte sequence number for keeping the cards in order (depending on the actual encoding format used).

so at ~700MB per CD, that’s 10,194,489 cards per CD, multiplied by 1200 CD’s is 1,223,386,800 punch cards which, at 56 cards/cm, would be a stack 2184km (1365 miles) high of machine-readable punch cards.

JoeCool (profile) says:

Re: Re: CDs?

Your punch cards had a sequence number? How ADVANCED they were! Ours had no such thing – if you got them out of order, you were sh_t out of luck. Funniest and saddest thing I ever saw was a grad student wheeling in a load of punch cards for his project (had to have been at least 10,000 cards) only for the cart to overturn spilling cards EVERYWHERE. The poor guy cried for almost an hour while picking up cards. I kept mine in bundles of 100 held together with rubber bands and a note telling me which group was which.

In fact, our card punch machines very often didn’t have an inked ribbon, so you had to read the cards from the punches. I got really good at that. Of course, I was in heaven when I finally got an English computer account. Yes, English students had a better account than the engineering students. You got to use the VAX instead of the AS9000, which meant actual terminals for programming instead of punch cards, and you got six hours of time as opposed to one. Weird, but I guess there were fewer English students using accounts, so they had more resources to split fewer ways.

Anonymous Coward says:

Re: Re: Quite small actually

I stand corrected. At least the file I viewed was a relatively low-res PDF image. Perhaps not quite designed to foil OCR, but definitely designed to require manual review and dissection.

Other features of note:
“collections”, but no information on whether the entire database has been filed in one (or more) collections, or whether there are items unindexed (save perhaps by ID number).

Nothing like a downloadable index into the archive. One might have success iterating through all possible ID numbers in order to see everything there is to see.

Celeste Guarini says:

CIA is in the news a lot lately. Maryland's databases too.

Must be part of a pre-emptive and self protective PR push- the CIA is anticipating other ‘leaks’ about the special material that can be found in it’s databases- and that material stretching from Thailand to Maryland, to Virginia like a big, baby eating HYDRA/octopus.

Try an FOIA on RAF Croughton and sex trafficking (and what did the British minister find there?)- that would bring interesting results.

Awww never mind- you might just flip out like “Michael Hastings.”

Anonymous Coward says:

‘plus ça change, plus c’est la même chose’ dept

[ ffwd 3:30 ]
The Hitchhikers Guide to the Galaxy Episode 1
Pedro Torres

“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”

“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”

“But look, you found the notice, didn’t you?”

“Yes,” said Arthur, “yes I did.

It was on display
in the bottom of a locked filing cabinet
in a disused lavatory
with a sign on the door
‘Beware of the Leopard.”

― Douglas Adams, The Hitchhiker’s Guide to the Galaxy

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...