There Is Simply No Scientific Backing For TSA's Behavioral Detection Program

from the hearsay-isn't-evidence dept

The Government Accountability Office has taken a run at the TSA's Behavioral Detection program in the past. Its findings were far from complimentary. Specially-trained "Behavior Detection Officers" (BDOs) were basically human coin flips. Deciding whether or not someone was a threat came down to a lot of subjective readings of human behavior, rather than proven principles.

In response to this report, the TSA started trimming back the number of BDOs it deployed, converting about 500 of them back into regular TSA officers. But the TSA still believed there was something to its pseudoscience patchwork, so it's still sending out 2,600 BDOs to covertly stare at travelers' throats and eyes (no, really) until terrorism reveals itself.

The GAO's second report focuses on the TSA's stubborn insistence that the Behavior Detection program is worth what we're paying for it. It has made claims to oversight that this program is scientifically-backed and scientifically-based. The GAO's investigation [PDF] finds almost nothing that backs these assertions.

We reviewed and categorized all 178 sources that, as of April 2017, TSA cited as providing support for specific indicators in its revised list of behavioral indicators to identify the extent to which they present valid evidence. We defined valid evidence as original research that meets generally accepted research standards and presents evidence that is applicable in supporting TSA’s specific behavioral indicators.

[...]

Of the 178 total sources TSA cited, we determined that 137 are news or opinion sources, and we took no further action because these do not meet our definition of valid evidence. We determined that 21 of the 178 sources are reviews of studies— sources that rely on the source author’s assertion of support for the indicator rather than original analysis, methods, or data that can be independently used as valid evidence.

Of the 178 sources cited by the TSA (with great confidence, one imagines), only 20 met the GAO's definition of valid evidence.

The TSA claims its BDOs have the ability to sniff out terrorism by looking for behavioral indicators. There are long lists of indicators to look for but, again, the TSA has nothing in the way of evidence to back these up.

In our review of all 178 sources TSA cited in support of its revised list, we found that 98 percent (175 of 178) of the sources do not provide valid evidence applicable to the specific indicators that TSA identified them as supporting. In total, we found that TSA does not have valid evidence to support 28 of its 36 revised behavioral indicators, has one source of valid evidence to support each of 7 indicators, and has 2 sources of valid evidence to support 1 indicator.

In fact, it appears the TSA may just be Googling for indicators, as its supporting "evidence" is extremely scattershot.

For example, among other sources, TSA cited four separate news articles that provide eyewitness accounts of a single 2005 suicide bombing incident at an Israeli bus station as support for 2 behavioral indicators. In another example, TSA cited a 2004 newsletter published by the New Hampshire Police Standards and Training Council as support for 11 behavioral indicators.

A partial snapshot of the chart created by the GAO shows the utter failure of the TSA's behavioral "science."

The individual breakdown of the scientific "support" for behavioral indicators is even more damning.

As a result of our analysis, we determined that TSA does not have valid evidence supporting 28 of its 36 revised behavioral indicators. Specifically, every source TSA cited in support of these 28 indicators is a news or opinion source, a review of studies source, or an original research source that either does not meet generally accepted research standards or is not applicable to the behavioral indicator it was provided to support. For example, TSA cited 105 sources to support the use of indicator number 11, which involves BDOs identifying individuals who seem to be attempting to conceal their normal appearance. However, we found that none of the 105 sources present original research that meets generally accepted research standards. In another example, TSA cited 63 sources to support the use of indicator number 5, which involves BDOs identifying individuals who seem to be sweating heavily. While we found that one of the 63 sources cited is original research that meets generally accepted research standards, we also found that this source does not present evidence that is applicable as support for this indicator.

To which is appended this footnote:

Some reviews of studies sources TSA cited as support for behavioral indicators present conflicting information on whether the use of behavioral indicators and behavior detection more generally is useful in identifying individuals who may pose a threat to security. For example, one article states that indicators of suicide bombers “are drawn from multiple sources and have not been formally or empirically validated.” Another states that indicators used by law enforcement to identify potential suicide bombers are “vague, contradictory, and so broad as to be useless.”

The DHS responded to the GAO's draft by claiming it still had plenty of research and science justifying the deployment of BDOs. More "evidence" was submitted and found to be just as useless as everything the GAO had already examined.

[I]n its letter, DHS cited passages from a 2013 RAND report as providing support for TSA’s use of behavior detection. However, the report DHS cited does not clearly support TSA’s use of behavior detection. As DHS noted, the RAND report states that there is value and unrealized potential for using behavioral indicators as part of a system to detect attacks. However, the indicators reviewed in the RAND study could not be used in real time in an airport environment, as we reported in 2013. Further, the RAND report refers to behavioral indicators that are defined and used significantly more broadly than those used by TSA. For example, the RAND report includes indicators such as mobile device tracking, monitoring online activity, and changes in lifestyle patterns.

The DHS also said the program was justified because trained BDOs also sometimes saw things even untrained non-professionals might be able to spot.

DHS also stated that there are certain common-sense indicators that TSA cannot reasonably ignore, with or without valid evidence. Specifically, DHS highlighted the indicator “unusual exposed wires or electrical switches on a person.”

Of course, these could (MAYBE) be detected by normal screeners, not specially-trained officers costing the taxpayers millions each year. The GAO agrees BDOs should ignore common sense. But much of what the DHS considers to be "common sense indicators" are highly-subjective takes on normal human behavior.

We recognize DHS’s position that certain common-sense indicators of mal-intent should not be ignored. However, TSA’s revised list of 36 behavioral indicators also includes indicators that are subjective, such as assessing the way an individual swallows or evaluating the degree to which an individual’s eyes are open.

Finally, the DHS protests it should be able to do it because other agencies (CBP, FBI) and other countries do it. The GAO patiently responds that behavioral detection may work for others, but the point is the TSA has yet to prove it works when it does it and can offer no solid evidence to back its claims.

The TSA's Behavioral Detection program isn't dead yet, but a few more nails have been driven into the coffin. If the regular TSA screening procedures are security theater, the BDOs are the mimes. A whole lot of moving around, but in the end, there's really nothing there.


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • icon
    DannyB (profile), 26 Jul 2017 @ 10:51am

    How about pseudo scientific backing?

    The TSA needs something to legitimize and rationalize what it wants to do. Racial profiling. Groping.

    Having a bunch of indicators you can check off gives it an air of legitimacy.

    reply to this | link to this | view in chronology ]

  • icon
    John Snape (profile), 26 Jul 2017 @ 10:51am

    I offer my services

    In light of recent happenings, I hereby offer my professional phrenologistic services to the TSA in hopes of advancing their anti-terroristic aims.

    I only request that I be recompensed $5,000 per month in hopes my struggles will make our skies 0.01% safer.

    reply to this | link to this | view in chronology ]

    • icon
      Bergman (profile), 26 Jul 2017 @ 11:59am

      Re: I offer my services

      What would be amusing is noticing that a TSA agent displays one or more traits that indicate suspicious behavior, according to the TSA's own procedures.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 26 Jul 2017 @ 10:54am

    Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

    Don't know where you get the notion that "scientific" must mean good. Just look at Nazi Germany: leader in technology, in legalizing all state actions, and corporatocracy, all "scientific".

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 26 Jul 2017 @ 10:55am

      Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

      science is not the same as application

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 26 Jul 2017 @ 11:04am

        Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

        @ "science is not the same as application"

        What's your point? "Scientific" is the word used in title, and means a method of approaching / solving problems.

        SHEESH. No matter how neutral and uncontroversial of comments, on this site full of contrarians you can be sure that some weenie will try to gainsay.

        reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 26 Jul 2017 @ 11:04am

        Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

        @ "science is not the same as application"

        What's your point? "Scientific" is the word used in title, and means a method of approaching / solving problems.

        SHEESH. No matter how neutral and uncontroversial of comments, on this site full of contrarians you can be sure that some weenie will try to gainsay.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 26 Jul 2017 @ 11:19am

          Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

          You both stubbed your toes.

          I agree with your question "What's your point?"
          "science is not the same as application" has no meaningful context.

          "SHEESH. No matter how neutral and uncontroversial of comments,..."

          Everything is uncontroversial until someone creates a controversy out of it.


          "... on this site full of contrarians you can be sure that some weenie will try to gainsay."

          This is the nature of social interaction. Someone usually has something to say about something. It is not bad to offer new ideas, even if they are stupid. If the ignorant appear, they become teachable moments. If knowledge is presented, then everyone has the opportunity for enrichment.

          reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 26 Jul 2017 @ 5:03pm

          Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

          I see you are not a scientist.

          reply to this | link to this | view in chronology ]

        • icon
          orbitalinsertion (profile), 26 Jul 2017 @ 5:47pm

          Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

          Yeah, and when it is pointed out how there is nothing scientific about any of it, one still has to make a comment about how 'scientific doesn't mean good'.

          Scientific does, however, mean scientific. Calling something scientific when it is, in fact, not, is either a lie or extremely sloppy thinking. Or both.

          It also does not help to confuse the scientific method (which is what "scientific" means here), and any products of a scientific endeavour, which may be used for good or ill, and in which the particular endeavour itself may have been a bad idea or outright nefarious to begin with.

          In that sense, scientific is not necessarily a good, but is also irrelevant when the subject at hand is not in the least scientific to begin with.

          Further, there is little which is scientific in the example given. If one wanted to specifically address Nazi sciences, some of them are entirely unscientific also, merely being labelled as science. Others are entirely common to any system, such as base R&D which might be applied to weapons systems, but most of the development is actually engineering issues. (But it's funny how i have never heard similar commentary for engineering as i have heard regarding science, or what someone wishes to label as science. Which is the sign of something being super trope-y and not particularly a reflection of reality.)

          Controversial? I don't care to speculate. But it is highly inaccurate. And there is at least one weenie somewhere who will always support some poorly reasoned statement or line of outright bullshit, no?

          reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 26 Jul 2017 @ 11:11pm

          Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

          site full of contrarians

          Pot, kettle.

          The irony burns.

          reply to this | link to this | view in chronology ]

    • icon
      Roger Strong (profile), 26 Jul 2017 @ 11:33am

      Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

      Don't know where you get the notion that "scientific" must mean good

      That'd come from a proper high-school science class. Where you're taught how to properly put a claim or theory to the test, with control groups and whatnot to avoid false conclusions and bias. Taught how to rule out - or acknowledge - other possible explanations. Taught to make your test repeatable by other independent testers to further rule out mistakes, deception and bias.

      That includes the TSA methods described above. There's no scientific backing for it. Nazis, "legalizing state actions" and corporatocracy have nothing to do with science. Technology is only vaguely related.

      You complain of "training the slaves to obey authority without question." Science class is all about teaching kids to question everything. Add a method for uncovering the truth and debunking garbage claims, and yes, it's a good thing.

      reply to this | link to this | view in chronology ]

      • icon
        JoeCool (profile), 26 Jul 2017 @ 3:03pm

        Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

        Science class is all about teaching kids to question everything.

        Well, that's what it SHOULD be about. In most cases, it's about regurgitating currently accepted "facts" no matter how ridiculous they seem when considered logically. Critical thinking is indeed NOT taught as then you have a generation of people capable of thinking for themselves, which makes them unsuitable to be cogs in the wheel.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 26 Jul 2017 @ 6:13pm

          Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

          it's about regurgitating currently accepted "facts" no matter how ridiculous they seem when considered logically.

          Like the so-called "theory of evolution". Anyone with half a brain can see how ridiculous that is.

          reply to this | link to this | view in chronology ]

          • identicon
            Anonymous Coward, 27 Jul 2017 @ 7:50am

            Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

            and let us not forget that Theory of Gravity, it is just a theory.

            reply to this | link to this | view in chronology ]

          • icon
            Roger Strong (profile), 27 Jul 2017 @ 8:05am

            Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

            Well, that's what the invisible magic sky fairy says.

            reply to this | link to this | view in chronology ]

          • icon
            JoeCool (profile), 29 Jul 2017 @ 11:52am

            Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

            You kinda read my post backwards. It's common in schools to NOT teach evolution, or to teach creationism along side evolution. This is what I mean when referring to "currently accepted 'facts'". That's also just a small example - history classes are almost devoid of real factual information and are almost completely composed of anecdotes written by popular fiction authors of the late 1800's and early 1900's.

            reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 26 Jul 2017 @ 10:54am

    Behavioral Detection Program sounds like the leo program that claims to be able to tell if someone is on drugs just by looking at them.

    reply to this | link to this | view in chronology ]

  • icon
    Anonymous Anonymous Coward (profile), 26 Jul 2017 @ 11:00am

    Self fulfilling

    Given TSA's recent behavior, isn't showing up at the airport a sign that something is demonstrably wrong with someone?

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 26 Jul 2017 @ 11:04am

    There is!

    "There Is Simply No Scientific Backing For TSA's Behavioral Detection Program"

    It just depends on which science you are using. Read up on science surrounding oppressing citizens and conditioning them to accept commands from any form of authority out of habit.

    The TSA is doing a fantastic job at obedience training. The people are lining up like cattle and subjecting themselves to every demand being made with little resistance.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 26 Jul 2017 @ 11:30am

      Re: There is!

      Or, like polygraphs, its main benefit may come in terms of intimidation. That could be scientifically studied.

      reply to this | link to this | view in chronology ]

  • identicon
    Christenson, 26 Jul 2017 @ 11:34am

    A different approach

    Given:
    *A ride on public transit is a low-cost thing to steal, (most systems incur near zero marginal cost per passenger until they have to increase capacity somewhere) and
    * public transit is generally subsidized, either by the fare-paying public or by taxes,

    How about not being too worried about a few free tickets to the enterprising or the poor who just might need them?

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 26 Jul 2017 @ 1:06pm

      Re: A different approach

      Except this is part of quality of life. NYC, in the eyes of quite a few people who live there, is going downhill, backwards. Now they just passed a law saying that people who urinate in the street won't be arrested. Fare jumpers won't be arrested.

      Who the hell wants to live in a place where that crap is allowed?

      reply to this | link to this | view in chronology ]

    • icon
      JoeCool (profile), 26 Jul 2017 @ 3:16pm

      Re: A different approach

      Wrong article. This is the TSA article, not the one on the Hungarian transit.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 26 Jul 2017 @ 11:36am

    Israel uses this. People getting on planes know they are being looked at, evaluated, if things seem out of sorts, they will talk to people, search them more thoroughly.

    I can see how this would be effective. People that are up to no good and know they are being watched tend to exhibit signs. Of course, some people may just be nervous and end up being delayed a bit, but whatever. When is the last time you heard of an El Al flight having a problem?

    Same with good cops, good police work involves watching people and seeing things that are outside the norm and then questioning. That is just good police work.

    reply to this | link to this | view in chronology ]

    • icon
      JoeCool (profile), 26 Jul 2017 @ 3:06pm

      Re:

      Citations needed...

      reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 26 Jul 2017 @ 4:03pm

      Re:

      When is the last time you heard of an El Al flight having a problem?

      It's extremely rare for any flight to have a security problem. We keep finding out it's really easy to get guns, bombs, etc. onto US flights, but when does it ever happen? The US and Israel spend a lot on security. Is there evidence they're more secure than countries that spend less?

      In lots of countries you can get on a train without any security check. When was the last time you heard of a train having a problem? All the ones I can think of were unintentional derailments.

      reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 26 Jul 2017 @ 5:07pm

      Re:

      "When is the last time you heard of an El Al flight having a problem?"

      This rock keeps tigers away ...

      reply to this | link to this | view in chronology ]

  • identicon
    Capt ICE Enforcer, 26 Jul 2017 @ 12:00pm

    Sorry

    I am sorry. But when has the government ever used science for any decision?

    reply to this | link to this | view in chronology ]

    • identicon
      Thad, 26 Jul 2017 @ 4:22pm

      Re: Sorry

      I'm pretty sure all those space shuttle launches had some science involved in them somewhere.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 26 Jul 2017 @ 6:20pm

        Re: Re: Sorry

        I'm pretty sure all those space shuttle launches had some science involved in them somewhere.

        Even the ones that blew up? Oh, wait, those were the ones where the bureaucrats overrode the engineers. Never mind.

        reply to this | link to this | view in chronology ]

  • icon
    Coyne Tibbets (profile), 26 Jul 2017 @ 6:43pm

    Snake oil

    I have to admit that their evidence is better than for most snake oil.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 26 Jul 2017 @ 7:42pm

    Brought to you by...

    ...the same people who "justify" illegal searches by saying, "my dog said so." This just cuts out the middle-man, er, um, dog.

    reply to this | link to this | view in chronology ]

  • identicon
    Meh, 27 Jul 2017 @ 10:45am

    Looks Like TSA watched too many episodes of Lie To Me.

    I wonder if any of their sources refer to the show as well?

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Insider Shop - Show Your Support!

Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.