US Government Admits: Data Mining For Terrorists Doesn't Work

from the oops dept

Over the last few years, you probably have noticed multiple attempts by state and federal government agencies to collect more data on people and to share that data more widely. This is troubling for a variety of reasons, often having to do with privacy and questions concerning the quality of data, but those concerns are almost always pushed aside by claims that this data is necessary for fighting terrorism. There's this myth out there that if we just had more data, somehow we'd be able to stop terrorists. The problem is that this is untrue. More data often makes it even harder to find the important data, and now a detailed government report has basically confirmed that data mining doesn't help in finding terrorists.

The report was put together by a bunch of well-respected academics, industry insiders and law enforcement officials on behalf of the National Research Council, and finds that the idea of plugging all this data into a machine and popping out terrorist identities is a myth. The report then lists out a variety of recommendations, concerning the use of any data mining systems, and how to make sure that individual privacy is protected. One key finding that will never be implemented: when using any such system, systematic reviews should be done to see if the systems are actually effective. If there's one thing we've seen with government programs over the years, such attempts to actually review something they've put in place to see if it actually worked are almost never done.


Reader Comments (rss)

(Flattened / Threaded)

  1.  
    identicon
    Anonymous Coward, Oct 7th, 2008 @ 11:56pm

    If our government only had the courage to engage skynet, then all that data could be analyzed well enough to ferret out the terrorists.. but I guess they aren't worried about terrorism. Maybe it plays into their hands?

     

    reply to this | link to this | view in thread ]

  2.  
    identicon
    Mr Big Content, Oct 8th, 2008 @ 12:25am

    Reality Has A Liberal Bias

    I don't buy all this "research" and "evidence" baloney. Look for the "truth", and you're always going to come up with a "true" conclusion. And what is "truth", but just another kind of namby-pamby pinko liberal bias?

    No, the way to come up with the right, God-fearing answer, is to leave no stone unturned in your determination to turn up that answer. If these researchers had the balls to do that, they would have seen the light, that what the Government is doing is the only sane way to make sure the TERRRISTS don't take over this country.

     

    reply to this | link to this | view in thread ]

  3.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 2:03am

    Re: Reality Has A Liberal Bias

    You must be a GenXer who doesn't get social networking sites.

     

    reply to this | link to this | view in thread ]

  4.  
    identicon
    oneofmany, Oct 8th, 2008 @ 2:06am

    no way!

    really? what a surprise! maybe it has to do with the fact that unlike those same academics and government overpaid employees the terrorist don't really use a computer all the time post on blogs and such. You can't data mine when you don't have digital data. Plus the effectiveness of data mining in many cases is quite controversial.

     

    reply to this | link to this | view in thread ]

  5.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 2:11am

    Re:

     

    reply to this | link to this | view in thread ]

  6.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 2:37am

    Failure will continue...

    ...because you can't model human behavior with math.

    They've tried this a few times- Enron was the first iteration (learning about what's necessary for business buy-in) The current financial collapse was the second iteration (where personal information was used to determine and mitigate credit risk)

    If this doesn't stop, and return to fundamentals... think about it.

    Who are "The smartest guys in the room"?

     

    reply to this | link to this | view in thread ]

  7.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 4:59am

    Re: Reality Has A Liberal Bias

    sarcasm?

     

    reply to this | link to this | view in thread ]

  8.  
    identicon
    pleasegetreal, Oct 8th, 2008 @ 5:10am

    really?

    Terrorists, we've decided that data mining for information regarding you doesn't work. Therefore, we aren't going to do it anymore. So don't worry about that anymore and carry on! Wink, wink.

     

    reply to this | link to this | view in thread ]

  9.  
    icon
    Isaac K (profile), Oct 8th, 2008 @ 5:34am

    Uhmn, as a government worker...

    If there's one thing we've seen with government programs over the years, such attempts to actually review something they've put in place to see if it actually worked are almost never done.

    I guess I shouldn't be surprised the general public doesn't know this, but the OIG and GAO are institutions created to specifically perform such reviews on a constant basis.

    And I assure you, they are VERY thorough. In my agency, the operating budget is less than 2% of our total outlays, basically unheard of anywhere, government or private, except for some non-profit orgs. Efficiency? Large government may not be the MOST efficient method, but a large part of the cost is the intervening layers filtering from a national system down to the local level. And that is where most of the problems, trickery, etc. come into play.

    All-in-all, I agree though; it would be very hard to determine the effectiveness of such a data filtration system.
    If test returns a negative result, it might just be that the data you are looking for doesn't exist, rather than a problem with the aggregator/filter.

     

    reply to this | link to this | view in thread ]

  10.  
    identicon
    T-1000, Oct 8th, 2008 @ 5:38am

    Do not worrry...

    >If our government only had the courage to engage skynet,
    >then all that data could be analyzed well enough to
    >ferret out the terrorists.. but I guess they aren't
    >worried about terrorism. Maybe it plays into their hands?

    Not to worry... SkyNet will engage itself!

     

    reply to this | link to this | view in thread ]

  11.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 5:43am

    This is a disgusting smear campaign against our troops. Nancy Pelosi and the Demo-facists are behind this and everyone knows it. If you techdirt hippies keep suporting stuff like this . . . well I hope your up on your farci, you may need it soon.


    VOTE McCain 2008 - He's gonna go all Maverick on Government!

     

    reply to this | link to this | view in thread ]

  12.  
    identicon
    NullOp, Oct 8th, 2008 @ 5:46am

    Under optimal conditions is almost impossible...

    If you have ever done any sort of survaillance you will know how much "junk" you come up with. Now, if you monitor "everything" you will realize how faint the sound of terrorism is in the static of everyday life/communications. In short, its a lot like listening for ET.

    BTW, yes, you can model human behavior with math. Just not as well as Wal-Mart would like to be able to do.

     

    reply to this | link to this | view in thread ]

  13.  
    identicon
    Vincent Clement, Oct 8th, 2008 @ 5:48am

    Next...

    Can we stop having to take our shoes off at airport security?

     

    reply to this | link to this | view in thread ]

  14.  
    identicon
    DanC, Oct 8th, 2008 @ 6:00am

    Re: Next...

    Nope, sorry. Even though data mining for terrorism doesn't work, the shoe bomb epidemic continues...

     

    reply to this | link to this | view in thread ]

  15.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 6:06am

    Re: Under optimal conditions is almost impossible...

     

    reply to this | link to this | view in thread ]

  16.  
    identicon
    Pelroston, Oct 8th, 2008 @ 6:07am

    Other Purposes

    ...data-mining is quite useful to the government for purposes 'other' than finding terrorists.

    Heavy surveillance of the domestic population (.. especially political opponents & non-cooperative citizens) is greatly aided by data-mining.

    Extending political control is always the objective of all government actors.

    Assuming that anti-terrorism is the only objective of data-mining... is a HUGE mistake.

     

    reply to this | link to this | view in thread ]

  17.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 6:23am

    Re: Re: Reality Has A Liberal Bias

    No.... he's a troll/satirist.

    You new to the internets AC?

     

    reply to this | link to this | view in thread ]

  18.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 6:36am

    I like big butts, and I can not lie. Will this show up in data mining?

     

    reply to this | link to this | view in thread ]

  19.  
    identicon
    Anonymous Coward, Oct 8th, 2008 @ 6:53am

    Re: Re:

    Hmm. That seems to be a problem- is that no computer can replace that which a human-to-human interaction has. Frankly, I don't care. I may have alternative political views to yours, but does that make me an enemy? Depends on who is in charge, I guess.

    Those that have met me know I have the best intentions, but what real emotion can be created and shared through a computer screen? It's always subject to the receptiveness to recieving side. If someone wants to defame me based on data mined political or social views, fine. Want to see who I donate to? Go for it. Chances are, you are probably close enough to me to just pick up the fucking phone and ask.

    You, sir, are the coward. So have fun watching me as I have fun exploring the world and interwebs. Maybe you'll learn something too.

     

    reply to this | link to this | view in thread ]

  20.  
    identicon
    Runa, Oct 27th, 2008 @ 12:37pm

    Good for people to know.

     

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This