The Stasi's Tiny Torn-Up Analog Files Defeat Modern Digital Technology's Attempts To Re-Assemble East Germany's Surveillance Records

from the too-hard-for-today's-hardware dept

It is nearly 30 years since the wall separating East and West Berlin came down, and yet work is still going on to deal with the toxic political legacy of East Germany. As Techdirt readers are well aware, one of the defining characteristics of the regime in East Germany was the unprecedented -- for the time, at least -- level of surveillance inflicted on citizens by the Stasi (short for Staatssicherheitsdienst, or State Security Service). This led to the creation of huge archives holding dossiers about millions of people.

As it became clear that East Germany's government would fall, and that its long-suffering citizens would demand to know who had been spying on them over the years, Stasi officers began to destroy the most incriminating documents. But there were so many files -- a 2008 Wired article about them says they occupied 100 miles of shelving -- that the shredding machines they used started to burn out. Eventually, Stasi agents were reduced to tearing pages by hand -- some 45 million of them, ripping them into around 600 million scraps of paper.

After thousands of bags holding the torn sheets were recovered, a team working for the Stasi records agency, the body responsible for handling the mountain of paper left behind by the secret police, began assembling the pages manually. It was hoped that the re-assembled documents would shed further light on the Stasi and its deeper secrets. But it was calculated that it would take 700 years to deal with all the scraps of paper by hand. A computerized approach was devised by the Fraunhofer Institute, best-known for devising the MP3 format, and implemented following a pilot project. After some initial successes, the program has run into problems, as the Guardian reports:

A so-called ePuzzler, working with an algorithm developed by the Fraunhofer Institute and costing about €8m of [German] federal funds, has managed to digitally reassemble about 91,000 pages since 2013. However, it has recently run into trouble.

For the last two years, the Stasi records agency has been waiting for engineers to develop more advanced hardware that can scan in smaller snippets, some of which are only the size of a fingernail.

The ePuzzler works by matching up types of paper stock, typewriter fonts, or the outline of the torn-up page. It has struggled with hand-written files that were folded before being torn, leaving several snippets with near-identical outlines.

While the hardware engineers try to come up with a suitable scanner that can handle these tiny fragments, a small team continues to match up the more crudely ripped pages manually. Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..."

Follow me @glynmoody on Twitter or identi.ca, and +glynmoody on Google+


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • This comment has been flagged by the community. Click here to show it
    identicon
    Anonymous Coward, 8 Jan 2018 @ 7:51pm

    "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

    Both Google and Facebook have BILLIONS OF PERSONS INDEXED. And you think it's good, merely "commercial", and that gov'ts are able to resist detailed data. -- You are the nebbishes in this sci-fi dystopia.


    "Inevitably, some people will be thinking: "If only the Stasi had used blockchain, all these problems could have been avoided..." -- ??? One of your wackiest attempts at "color". Or whatever.

    reply to this | link to this | view in chronology ]

    • icon
      steell (profile), 8 Jan 2018 @ 8:46pm

      Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

      Dude, generally I refrain from replying to your posts, I just flag them and move on. But this one is posted from the depths of ignorance so great the bottom may never be found.

      So tell me oh ignorant one, does Google make people disappear? Does Google slip a note in your pocket saying if you don't cooperate all your transgressions will be revealed to Command? Does Google threaten your family with harm if you don't comply?

      To compare Google to the Stasi is the stupidest thing I've ever read.

      reply to this | link to this | view in chronology ]

      • icon
        Richard (profile), 9 Jan 2018 @ 3:59am

        Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

        To compare Google to the Stasi is the stupidest thing I've ever read.

        It is not true that comparing Google to the Stasi is stupid.

        (even though his comparison was)

        Both collect(ed) large amounts of personal information.

        Google uses it for purposes that are not necessarily evil, and may sometimes be beneficial to the subject.

        The risk with Google is a 3rd (stasi -like) party getting access to the information and using it in an evil way.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 9 Jan 2018 @ 4:06am

          Re: Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

          I suppose you could claim the goog is a stooge but the staz?
          I dont think so.

          reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jan 2018 @ 1:22pm

        Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

        "Does Google slip a note in your pocket saying if you don't cooperate all your transgressions will be revealed to Command?"

        I would categorize ad-tracking as being within the same wheel house. Perhaps, not to "command", but to any advertiser willing to pay.

        Targeted advertising supplied by Google has probably outed more homosexuals than all the angry ex-girlfriends on the planet. Not to mention outing peoples medical conditions, and just about anything else a person might be embarrassed about.

        The Stassi at least charged a serious bribe. Google outs people for fractions of a cent. Notably TD contributes to this since they use Googles analytics services.

        Or do you not consider the causality of the TV advertising you see in other peoples homes?

        reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 8 Jan 2018 @ 9:32pm

      Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

      And yet you suck the cock of the NSA at every given opportunity because they help the RIAA with suing pirates.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jan 2018 @ 3:01am

        Re: Re: "unprecedented -- for the time, at least --" but nothing compared to Google and Facebook.

        Well, fairness, he also sucks Trump's rancid pus-oozing cock as well. Equal time, you know. ;)

        reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 8 Jan 2018 @ 9:32pm

    Compare this...

    reply to this | link to this | view in chronology ]

  • identicon
    Agammamon, 8 Jan 2018 @ 10:31pm

    "Eventually, Stasi agents were reduced to tearing pages by hand"

    And at no point did any of them think 'this would be a lot easier with a couple gallons of petrol and a pack of matches'?

    reply to this | link to this | view in chronology ]

    • icon
      That One Guy (profile), 8 Jan 2018 @ 11:14pm

      Re:

      I was thinking that same thing. Did they not realize that 'fire' was a thing? A bonfire or 'accidental' building fire would have cleared the whole thing up real quick, yet they spent I don't even want to think how much time manually ripping up pages of highly flammable paper.

      The Stasi: Great at collecting evidence, idiots when it comes to destroying it.

      Still, I suppose people should be glad they were stupid in that way, as there's at least a chance to reconstruct ripped up pages, a possibility that doesn't work so well when it comes to ashes.

      reply to this | link to this | view in chronology ]

      • icon
        Richard (profile), 9 Jan 2018 @ 3:18am

        Re: Re:

        Not burning the files is all part of their evil plan. When the German economy is flattened by all the effort deciphering them they can foment a new revolution and usher back a glorious communist future...

        reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 9 Jan 2018 @ 4:21am

        Re: Re:

        Take a look at Nazi practices: they murdered millions in remote and isolate places. But not all places were remote, some even really close to villages, for example the psychiatry in Hadamar. The fetidness from the included crematory was very well discernible by villagers (they all played along anyway).

        Likewise, a bonfire destroying the files would have sent a very clear signal, while Stasi was all about secrecy. Maybe Stasi officers weren't motivated anymore, which leads to worse performance at work. Or maybe they just couldn't afford the gas and/or the matches.

        reply to this | link to this | view in chronology ]

      • icon
        Toom1275 (profile), 9 Jan 2018 @ 6:24am

        Re: Re:

        Depends on the ashes. If they aren't burned properly and stirred afterward, you can get largish scraps of merely charred paper on which the ink is still fairly legible.

        reply to this | link to this | view in chronology ]

    • identicon
      Daydream, 9 Jan 2018 @ 2:50am

      Re:

      As I was reading the article, I actually considered that a bucket of water might have worked. Swish the paper around in the water until it's disintegrated. Or did they use water resistant paper?

      reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 9 Jan 2018 @ 6:21pm

      Re:

      And at no point did any of them think 'this would be a lot easier with a couple gallons of petrol and a pack of matches'?

      To quote the Wired article linked by Glyn, "Burning all those files would tip off angry Berliners that something was up."

      They were outnumbered, and worried about what would happen if the public found out what they had been doing. They wanted the most incrimining stuff destroyed before anyone found out they were destroying it.

      reply to this | link to this | view in chronology ]

  • icon
    Ben (profile), 8 Jan 2018 @ 11:53pm

    Book scanning writ large

    In Vernor Vinge's Rainbow's End there was a (somewhat) horrifying mechanism for scanning books: toss the books in a shredder and while the shredded pieces are floating down an air-filled channel they are photographed and then digitally reassembled. It certainly gets away from having to carefully scan each piece; I just wonder if the technique would require the re-shredding (further shredding?) of the Stasi papers to deal with the less than uniform existing pieces.

    Once a workable solution is found, however, this will have very big implications for the SEC and FBI trying to recover documents after corporate shredding parties.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 9 Jan 2018 @ 3:07am

      Re: Book scanning writ large

      To address that last point: maybe. Files are increasingly kept in digital form, which makes them much easier to disappear, provided those doing so have a baseline level of competence.

      For example, as part of my work, I sometimes decommission servers and thus the disks inside them. My modus operandi is to hook those disks up, one at a time, via SATA-to-USB hardware, to a system that runs quite a random write passes on the entire drive. Then I open them up and expose the platters to a degausser. Then I remove the platters and feed them to a shredder. Then I separate the the shreds and dispose of them in multiple different places.

      Good luck recovering anything from that, even with the resources of an intelligence agency. (Of course, if they were interested, they likely snagged all the data while the systems in question were still live.) The point is, though, that anyone who's trying to destroy data thoroughly has the means to do so, and it's much easier than dealing with miles of (paper) files.

      reply to this | link to this | view in chronology ]

      • icon
        Richard (profile), 9 Jan 2018 @ 3:20am

        Re: Re: Book scanning writ large

        Standard procedure in secure military circles is to etch the magnetic medium off the platters with a suitable solvent.

        reply to this | link to this | view in chronology ]

        • icon
          XcOM987 (profile), 9 Jan 2018 @ 4:38am

          Re: Re: Re: Book scanning writ large

          I just used to put a 6" nail through all the platters to shatter them before they went off to WEEE centers to be shredded. wouldln't stop someone seriously wanting the data, but would stop the opertune thief.

          reply to this | link to this | view in chronology ]

      • icon
        Ben (profile), 9 Jan 2018 @ 7:08am

        Re: Re: Book scanning writ large

        In the modern era, yes, all the data/memos would be on servers, so getting rid of it is much easier than in the past (and easier to explain away -- oh! that project ended so we decommissioned the server with a sledgehammer...)

        The issue is the so-called "paperless office." The reality is people still print things out -- it is just that they can re-print it at need. That is what they have shredding parties for.

        reply to this | link to this | view in chronology ]

      • icon
        The Wanderer (profile), 9 Jan 2018 @ 8:30am

        Re: Re: Book scanning writ large

        Even that isn't theoretically 100% guaranteed to render the data unrecoverable. It's enough for the limits of modern technology and the resources of pretty much any organization that exists, but in theory, a sufficiently-powerful organization could collect the scraps and shuffle through them and reassemble them (or scan them and reassemble the scanned data objects) and read the result.

        If you really want to guarantee that no one will ever be able to read what used to be on a given hard drive, you need to pull out the platters and melt them, then stir the result together till it's unrecognizable.

        Any entity capable of recovering the data from that would be so far into "indistinguishable from magic" that it could just as readily recover the data without access to the hard drive in the first place.

        reply to this | link to this | view in chronology ]

      • identicon
        TRX, 9 Jan 2018 @ 11:51am

        Re: Re: Book scanning writ large

        In the late 1980s engineers from IBM's Atlanta facility used to decommission hard drives by taking them to an indoor shooting range. The management didn't care as long as they swept up the mess afterward.

        Half a dozen rounds of .357 Magnum would make it difficult to recover any data...

        reply to this | link to this | view in chronology ]

  • icon
    scotts13 (profile), 9 Jan 2018 @ 4:59am

    Why?

    Call me naive, but wouldn't everyone be much better off if the files were finally and utterly destroyed? I understand a morbid curiosity, but as long as they exist, someone may yet use the files in an unfavorable manner. Start fresh, with a level playing field.

    reply to this | link to this | view in chronology ]

    • icon
      JoeCool (profile), 9 Jan 2018 @ 6:23am

      Re: Why?

      The papers are evidence of crimes. Until they are all reassembled and examined by police, you can't be sure you caught all the criminals. After their worth as evidence is expended, then you can dispose of them properly.

      reply to this | link to this | view in chronology ]

  • identicon
    Darkhog, 12 Jan 2018 @ 7:16pm

    You gotta admit that those Stasi guys were bunch of morons

    All they really had to do was to make a huge bonfire - good look recovering documents from ashes.

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: Techdirt Logo Gear
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.