TSA's $1 Billion 'Behavioral Detection' Program Only Slightly More Accurate Than A Coin Flip

from the treading-very-expensive-water dept

I have yet to see a Government Accountability Office report that can't be described as "highly negative" -- at minimum. Most tend to fall somewhere between "brutal" and "scathing." This is a good thing, although I imagine any agency that is the subject of these reports strongly feels otherwise.

Unfortunately, I'm also inclined to believe that I'll go to my grave before I see a GAO report giving its subject a thumbs up or a 3-out-of-5 star rating. This is a bad thing, because it suggests many of these agencies coast from scathing review to scathing review with minimal interest in improving things.

The GAO's report on the TSA's Behavioral Detection program, SPOT (Screening of Passengers by Observation Techniques), is out and it's every bit as bad as the laborious lead-in above would have you believe. First off, any program that immediately conjures up this scene from It's Always Sunny in Philadelphia is a program that has some serious problems.



[relevant dialog here for those unwilling/unable to view video]
Frank: Yeah, he probably just carved up his mother and father and he’s runnin’ away on the road.
Dennis: That’s right.
Mac: Could be, Frank. All I’m saying is that I’m ready for any situation. And that’s why you’re all safe as long as I’m around.
Dennis: …Uh-huh.
Mac: Yeah, I did an ocular pat-down and I cleared him.
Frank: Say what?
Dee: I’m sorry?
Mac: I’m saying that I did an ocular assessment of the situation, garnered that he was not a security risk and I cleared him for passage.
Dennis: What in the hell are you talking about?
Mac: I’m talking about breaking down the security situation, clearing an individual and making it safe for passage.
The TSA has been attempting for years to up its security game. Rather than roll back questionable barn-door-closing policies, it has instead opted to increase the number of highly trained BDOs (Behavioral Detection Officers) deployed in our nation's airports. This has been slowly developing since 2007 and hasn't improved over the last half-decade. All told, nearly a billion dollars have been sunk into the program, with the end result being a scattershot system that the GAO says detects potential threats with an accuracy that is "the same or slightly better than chance."

There are 99 pages full of investigative info and proposed remedies in the report, but most of what's wrong with the program is handled up front.

Four years after the rollout, a Technical Advisory Committee (TAC) was assembled to test the program's accuracy by comparing SPOT's hit rate with randomly selected passengers. But any hopes of this test providing an accurate reflection of the program's efficacy was completely undermined by the TSA's refusal to share any pertinent data with the TAC.
According to the TAC report, TAC members received briefings from the contractor that described the study plans and results, but because of TSA’s security concerns, TAC members did not receive detailed information about the contents of the SPOT referral report, the individual indicators used in the SPOT program, the validation study data, or the final report containing complete details of the SPOT validation study results. The TAC report noted that several TAC members felt that these restrictions hampered their ability to perform their assigned tasks.
In other words, the TSA was so "concerned" about security that it prevented a committee tasked with evaluating its security program from performing its duties. And, obviously, the overriding concern was for the TSA's security, not the passengers' security. The stymied TAC still produced a report, but considering the number of limitations it was subjected to, its findings were relatively meaningless.

The Behavioral Detection program was supposed to be the cutting edge of airport security, turning the agency into a smarter, more responsive security force. But instead of applying any sort of scientifically proven methods, the agency leaned heavily on BDO hunches. (This has been observed in other DHS agencies as well. The DHS notably stated in its Civil Liberties Assessment that CBP agent hunches should supersede any civil liberties concerns when it comes to searching electronics at the borders.)

The GAO has the following to say about the "science" behind the SPOT program.
Peer-reviewed, published research does not support whether the use of nonverbal behavioral indicators by human observers can accurately identify deception. Our review of meta-analyses and other studies related to detecting deception conducted over the past 60 years, and interviews with experts in the field, question the use of behavior observation techniques, that is, human observation unaided by technology, as a means for reliably detecting deception. The meta-analyses, or reviews that synthesize the findings of other studies, we reviewed collectively included research from more than 400 separate studies on detecting deception, and found that the ability of human observers to accurately identify deceptive behavior based on behavioral cues or indicators is the same as or slightly better than chance (54 percent).
There's more:
Moreover, a 2008 study performed for the Department of Defense by the JASON Program Office reviewed behavior detection programs, including the methods used by the SPOT program, and found that no compelling evidence exists to support remote observation of physiological signals that may indicate fear or nervousness in an operational scenario by human observers, and no scientific evidence exists to support the use of these signals in detecting or inferring future behavior or intent
Fear or nervousness is a common reaction in people with "nothing to hide" when trapped in a conversation with a TSA agent who has the power to do anything from preventing them from making their flight to holding them indefinitely and/or turning them over to law enforcement simply because they don't like an answer or the manner in which the answer was delivered.

And it goes far beyond simply sniffing around aimlessly for "suspicious behavior."
In particular, the report stated that success in identifying deception and intent in other studies is post hoc and such studies incorrectly equate success in identifying terrorists with the identification of drug smugglers, warrant violators, or others. For example, when describing the techniques used by BDOs in the SPOT program, the report concluded that even if a correlation were found between abnormal behaviors and guilt as a result of some transgression, there is no clear indication that the guilt caused the abnormal behavior. The report also noted that the determination that the abnormal behavior was caused by guilt was made after the fact, rather than being based on established criteria beforehand.
So, we have hunches and post facto attributions masquerading as a behavioral detection program. A billion dollars have been sunk into a program that has no scientific basis and that has never been conclusively proven to provide better threat detection than a BDO could achieve by flipping a coin.

This is what's supposedly keeping us safe from airborne acts of terrorism. Despite the lack of results, the TSA as recently as 2011 declared it was looking to expand this program. Sure, claiming to be following the "Israeli model" may make the agency look smarter, but in practice, there's very little indication this program has done anything to make BDOs better… or us safer.



Reader Comments (rss)

(Flattened / Threaded)

  1.  
    icon
    DannyB (profile), Nov 15th, 2013 @ 7:54am

    It is worth $1 Billion

    Look Tim,

    While this billion dollar program may be only slightly more accurate, at least it does not infringe upon my patent.

    My patent is for a method and system for making binary decisions based on the launching of a flat round decision support device into the air and making a determination of the outcome based on which side the decision support device lands on.

    I will also sell these decision support devices. A basic model for $10 is made of copper and is decorated with a picture of Lincoln on one side. A more expensive $25 model has a picture of George Washington and is constructed using superior metals.

    This is a valuable patent from which I anticipate making a mint (no pun intended).

    This is NOT a lame software patent. This is a patent on genuine hardware contributing a genuine advance in the important field of executive management decision making which has major applications in the areas of business, commerce, sporting events and terrorist detection.

     

    reply to this | link to this | view in thread ]

  2.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 7:57am

    Let me try an RIAA / MPAA argument here . . .

    If the TSA didn't spend that $1 Billion on this program, then that money would just disappear out of the economy!

     

    reply to this | link to this | view in thread ]

  3.  
    icon
    DannyB (profile), Nov 15th, 2013 @ 7:57am

    Re: Let me try an RIAA / MPAA argument here . . .

    That one is mine, but I wasn't signed in, for some strange reason.

     

    reply to this | link to this | view in thread ]

  4.  
    icon
    Not an Electronic Rodent (profile), Nov 15th, 2013 @ 8:16am

    Coming soon to an appropriations committee near you

    question the use of behavior observation techniques, that is, human observation unaided by technology, as a means for reliably detecting deception. (Emphasis mine)
    TSA: "Clearly we need an extra $3Bn in our budget to get the technology to go with our highly trained observers because, well, terrorists."

    Of course it's not about the money, it's for security, right?

     

    reply to this | link to this | view in thread ]

  5.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 8:16am

    Let me go on record with a prediction

    If and when the next substantial terrorist attack against aviation is carried out, it will include either (a) active participation of TSA personnel who have been bribed or blackmailed or (b) passive participation of TSA personnel who have been socially engineered into supporting it. I wouldn't be surprised if it's both.

    The TSA has spent the past decade-plus making travel LESS secure, stealing from travelers, sexually assaulting travelers, harassing travelers, and -- as we see every time someone bothers to check -- doing an absolutely horrible job with the basics of security. The best thing that could happen at this point would be the immediate and complete shutdown of the entire agency along with lifetime blacklisting for the idiots it has employed: those people should never be allowed to work again at ANYTHING more critical to society than a fast-food job.

     

    reply to this | link to this | view in thread ]

  6.  
    icon
    Carl "Bear" Bussjaeger (profile), Nov 15th, 2013 @ 8:36am

    If they've nothing to hide, they have nothing to fear in releasing the SPOT data

    I just love the excuse that, "We can't tell you what or how we do what we do because that would hurt what or how we do what we claim to do;" whether it's the TSA hiding their SPOT program, or the NSA and its spying, or...[insert agency/clusterf-k ad infinitum]... I carry a handgun for personal defense. The fact that ballistics data on the .45ACP cartridge has been widely published for a century doesn't seem to be hampering its effectiveness.

    But then, I'm concerned with personal security. They are concerned with their job security if they ever admit how screwed up and useless they are. Which, considering how tough it is to eliminate a federal program, just demonstrates how paranoid they are.

     

    reply to this | link to this | view in thread ]

  7.  
    icon
    Wally (profile), Nov 15th, 2013 @ 8:45am

    Man if this doesn't show the TSA's misappropriation of money to the public..I couldn't tell you what would. I watched the report about this on the news and CBS did an entire 48Hours segment on it.

     

    reply to this | link to this | view in thread ]

  8.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 9:05am

    They could've saved money and just flipped a penny instead. Sure, they'd be missing 4%, but is 4% really worth at least $999,999,999.99?

     

    reply to this | link to this | view in thread ]

  9.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 9:24am

    instead of trying to save this incredible waste of cash, why not go the whole 9 yards and try to save $2billion? do away with this arrogant bunch and get planes flying on time again, with minimal intrusion and interruption? how many actually disasters has this lot really stopped? about as many as the NSA spying, i suspect, ie none!

     

    reply to this | link to this | view in thread ]

  10.  
    identicon
    Anon, Nov 15th, 2013 @ 9:25am

    Seriously, Why?

    The number of incidents that hit the news appear to be about once a week or once a month. A few million people fly every month and one causes a problem.

    So even a .1% error rate means you harass and inconvenience, and likely destroy the travel plans, of a few thousand people a month for what - the one guy who caused an issue... Not hijacked a plane, but caused an issue.

    Of course, more than half of those problems are difficult drunks, so they do not get caught until after the plane is in the air.

    Serioussly, there hasn't been a problem since the shoe bomber and the underwear bomber - both of whom got on the plane outside the USA. So really, there has not been an airline threat that the TSA has had to find in the last 12 years.

    "Security Theatre" pretty much describes this. More likely, "security theatre of the absurd".

     

    reply to this | link to this | view in thread ]

  11.  
    icon
    John Fenderson (profile), Nov 15th, 2013 @ 9:57am

    Learning the wrong thing from cops

    Fear or nervousness is a common reaction in people with "nothing to hide" when trapped in a conversation with a TSA agent


    They apparently learned that indications of nervousness are indicative of guilt from cops, who seem to believe this to their bones.

    Interacting with cops is inherently terrifying (made intentionally so by the intimidating behavior of the cops). When I have had to interact with them, I have found it impossible to hide my fear, and they always ask something like "what are you so nervous about?", despite the answer being obvious: I'm talking to a cop.

     

    reply to this | link to this | view in thread ]

  12.  
    icon
    blaktron (profile), Nov 15th, 2013 @ 10:18am

    If the TSA agents could reliably detect deception, none of them would work for the TSA and would play professional poker instead.

     

    reply to this | link to this | view in thread ]

  13.  
    icon
    nasch (profile), Nov 15th, 2013 @ 10:40am

    Re: Let me go on record with a prediction

    If and when the next substantial terrorist attack against aviation is carried out, it will include either

    Could be c) detonation of a bomb in a security checkpoint line. You think air travel is a mess now, wait until the TSA overreact to that one...

     

    reply to this | link to this | view in thread ]

  14.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 10:42am

    Re: It is worth $1 Billion

    Excellent! I have a few hundred devices similar to your copper decision-making devices (from a previous production run) I'd like to invest in your company as soon as the patent is issued. Do let me know, please.

    P.S., I hope this does not constitute prior art.

     

    reply to this | link to this | view in thread ]

  15.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 10:59am

    Re:

    Maybe that's the answer. Fire all current agents and hire six successful, professional gamblers, three male, three female - one each per shift, per station. A casual glance at the flyers will tell them who's gonna give them a problem. One each shift will act a supervisor for the other, that position swapping each shift. Reports? We don't need no stinkin' reports...except in the highly unlikely event that something actually happens. This will save Billions. Oh, and you might want to actually TRAIN these new agents. They're probably smart enough to learn, unlike their predecessors.
    .

     

    reply to this | link to this | view in thread ]

  16.  
    icon
    aldestrawk (profile), Nov 15th, 2013 @ 12:03pm

    I don't think it is a fatal flaw for TSA to operate on a hunch. It would be extremely difficult to prove the efficacy of any such program for the simple reason that terrorist incidents are so rare. I have not read the report yet but I am rather skeptical about how they came up with a 54% effectiveness value. It's the same problem. Not enough data points for statistical accuracy.
    It simply not the same to have actors, playing the part of terrorists, testing the SPOT program to increase the number of data points. A hunch may be based on related scientific evidence and anecdotal reports. I think that is OK. What I am completely puzzled by is how a billion dollars was spent on this. I would expect perhaps a few million, mostly for training. The other main problem with it is acceptance by the public. I once went through a 25 minute interrogation by some young woman who was in the Israeli Army at the airport in Tel Aviv. I barely tolerated it because I was surprised by being picked out of the crowd of passengers and the fact that this was THE foreign nation who was noted for being heavy on airport security. I don't think I would tolerate this in the US. Their suspicions may have been encouraged by the fact that I was an American, living in Hungary with a passport issued in Spain, and traveling with a Belgium woman to whom I was not married. Oh, and we had just gotten back from Egypt. Still, the intrusion into our personal life I felt was unwarranted.

     

    reply to this | link to this | view in thread ]

  17.  
    icon
    John Fenderson (profile), Nov 15th, 2013 @ 12:05pm

    Re: Re:

    That won't work, either.

    The fact is that it's impossible to reliably tell when strangers are lying. You certainly can't tell by observing them, and there is no feasible machine or technology that can do it, either.

    Poker players are no better at telling when a stranger is being deceptive than anyone else. What they do is carefully watch the other players for a while so they can discover the unique behaviors that that particular player has that can indicate deception. That's what they mean by learning someone's "tell". However, that takes time and interaction and once they know a person's tell, that person is no longer a stranger.

     

    reply to this | link to this | view in thread ]

  18.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 12:05pm

    Re:

    You are terribly far off with your math there. You'd need at least one penny per airport. According to wikipedia, that's $1.62 worth of pennies you'd need. At that price, you might as well spend the rest to get that 4%.

    How's your math working out for you now?

     

    reply to this | link to this | view in thread ]

  19.  
    icon
    John Fenderson (profile), Nov 15th, 2013 @ 12:09pm

    Re:

    A hunch may be based on related scientific evidence and anecdotal reports.


    A hunch based on actual evidence isn't a hunch, it's a hypothesis.

     

    reply to this | link to this | view in thread ]

  20.  
    icon
    DannyB (profile), Nov 15th, 2013 @ 12:10pm

    Re: Re: Let me go on record with a prediction

    If a bomb goes off in a security checkpoint line, they will just create a new multi-Billion dollar screening system you must go through in order to get into the regular TSA security line.

    That way more people get to grope you.

     

    reply to this | link to this | view in thread ]

  21.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 12:14pm

    Re:

    The number didn't come from the TSA data. Remember, they wouldn't give it. Instead, the GAO got that number from looking at all the other studies that have tried to determine if behavior can be used to identify deception. This has been studied for decades. After decades, nobody has shown that it's possible. We're not talking about whether it's possible to pick out a terrorist, but any deception. And the studies show it's not any better than a coin flip.

    Here's the relevant quote:
    Peer-reviewed, published research does not support whether the use of nonverbal behavioral indicators by human observers can accurately identify deception. Our review of meta-analyses and other studies related to detecting deception conducted over the past 60 years, and interviews with experts in the field, question the use of behavior observation techniques...

     

    reply to this | link to this | view in thread ]

  22.  
    icon
    Dirkmaster (profile), Nov 15th, 2013 @ 12:26pm

    The Only security

    The only security that the TSA is concerned with is their job security. Their lack of cooperation with a test to study a programs' effectiveness is a perfect example of that.

    As it turns out, alas, even a bad report doesn't hurt them.

     

    reply to this | link to this | view in thread ]

  23.  
    icon
    Jeffrey Nonken (profile), Nov 15th, 2013 @ 12:26pm

    So... slightly better than a lie detector test, then?

    And not quite as good as astrology.

     

    reply to this | link to this | view in thread ]

  24.  
    icon
    That One Guy (profile), Nov 15th, 2013 @ 12:35pm

    Re: Re: Re: Let me go on record with a prediction

    If a bomb goes off in a security checkpoint line, they will just create a new multi-Billion dollar screening system you must go through in order to get into the regular TSA security line.

    Creating yet another line/target, completely ignoring what happened the first time... and the circle of stupidity continues.

     

    reply to this | link to this | view in thread ]

  25.  
    icon
    nasch (profile), Nov 15th, 2013 @ 1:02pm

    Re: Re: Re: Re: Let me go on record with a prediction

    and the circle of stupidity continues.

    It's the circle of duuuuuummbb, it's the wheel of stuuuupiiiiid....

     

    reply to this | link to this | view in thread ]

  26.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 1:02pm

    Behavioral Detection

    Everyone knows the best behavioral detection program is grandma.
    I never got away with shit, it was like she new what I was going to do before I did.

     

    reply to this | link to this | view in thread ]

  27.  
    icon
    John Fenderson (profile), Nov 15th, 2013 @ 1:06pm

    Re: Re: Re: Re: Let me go on record with a prediction

    Infinite recursion powers, activate!

     

    reply to this | link to this | view in thread ]

  28.  
    icon
    blaktron (profile), Nov 15th, 2013 @ 1:14pm

    Re: Re: Re:

    Exactly. Which is why if the TSA had some mystic training technique that could apply 'tell theory' to total strangers, those people almost universally qualify to cash in the Main Event.

    They also would have very few personal problems, you know, because you can't lie to them. All in all, the TSA would be creating supermen!

     

    reply to this | link to this | view in thread ]

  29.  
    icon
    aldestrawk (profile), Nov 15th, 2013 @ 2:22pm

    Re: Re:

    A hypothesis need not be based on evidence, you just need evidence to test it. It might be based on some existing evidence but it's not necessary. I see a hunch as a weak hypothesis.
    An example of a hypothesis based on a theory that you guess is applicable to something else, is applying the earthquake swarm theory of self-exciting points to crime prediction. Something that Techdirt dealt talked about a couple of weeks ago.

     

    reply to this | link to this | view in thread ]

  30.  
    icon
    beltorak (profile), Nov 15th, 2013 @ 2:28pm

    Re: Re: Re:

    So the solution is simple! Give the TSA access to the NSA databases! That way they will know everybody!!

     

    reply to this | link to this | view in thread ]

  31.  
    icon
    aldestrawk (profile), Nov 15th, 2013 @ 2:28pm

    Re: Re:

    Thanks for clarifying that. I am not surprised about the difficulty of detecting any sort of deception. What should be much easier is detecting a terrorist who is on a suicide mission. It is also the sort of thing that is so rare it is hard to study.

     

    reply to this | link to this | view in thread ]

  32.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 3:28pm

    Re: Re:

    Yeah, but all them $100 words are way beyond Bubba at the counter.

     

    reply to this | link to this | view in thread ]

  33.  
    identicon
    Anonymous Coward, Nov 15th, 2013 @ 3:58pm

    Re: Behavioral Detection

    Ah, Grandmothers. They know all, they see all, they hear all, even three rooms away. They've spent a lifetime learning the ways of boys, first your father then you. Grandpa, meanwhile, has spent his life perfecting his Sgt. Schultz act. They would be perfect in this role.

     

    reply to this | link to this | view in thread ]

  34.  
    icon
    timmaguire42 (profile), Nov 16th, 2013 @ 3:32am

    Al this analysis stuff is nice, but it's beside the point

    Just like the drug courier profile that amounts to "a black guy in a nice car driving the speed limit" or the airport profile that is little more than a grab-bag of traits and habits that all of us have to some degree, these things aren't about objectively assessing a situation and identifying the dangerous characters. They are about wrapping an agent's hunches in a veneer of science to protect the system from criticism.

    It's not really science? So what, it wasn't meant to be.

    Call it "Science Theatre".

     

    reply to this | link to this | view in thread ]

  35.  
    identicon
    Angry Citizen, Nov 17th, 2013 @ 6:08am

    Security Theater

    The planes are still cleaned by illegal aliens.

    Many of the TSA agents are convicted felons - even rapists and child molesters.

    That's government for you.

     

    reply to this | link to this | view in thread ]

  36.  
    identicon
    Dave, Nov 17th, 2013 @ 8:33am

    Re: Let me go on record with a prediction

    Given the fact that it was made public when TSA first infested America that the gov't was hiring an incredibly high (for 'security' work) percentage of convicted felons as new TSA staff, I think your prediction is a no-brainer. The number was in the thousands as I recall. You could be wrong about the use of the words "next substantial" but TSA complicity in an attack will certainly happen. They say that everybody can be bought at some price - but for ex-cons it is probably much cheaper than buying, for example, me.

     

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This