Studies

by Tim Cushing


Filed Under:
crime lab, dna, evidence, police, study



Study Buried For Four Years Shows Crime Lab DNA Testing Is Severely Flawed

from the doing-what-works-rather-than-doing-it-right dept

DNA is supposed to be the gold standard of evidence. Supposedly so distinct it would be impossible to convict the wrong person, yet DNA evidence has been given far more credit than it's earned.

Part of the problem is that it's indecipherable to laypeople. That has allowed crime lab technicians to testify to a level of certainty that's not backed by the data. Another, much larger problem is the testing itself. It searches for DNA matches in samples covered with unrelated DNA. Contamination is all but assured. In one stunning example of DNA testing's flaws, European law enforcement spent years chasing a nonexistent serial killer whose DNA was scattered across several crime scenes before coming to the realization the DNA officers kept finding belonged to the person packaging the testing swabs used by investigators.

The reputation of DNA testing remains mostly untainted, rose-tinted by the mental imagery of white-coated techs working in spotless labs to deliver justice, surrounded by all sorts of science stuff and high-powered computers. In reality, testing methods vary greatly from crime lab to crime lab, as do the standards for declaring a match. People lose their freedom thanks to inexact science and careless handling of samples. And it happens far more frequently than anyone involved in crime lab testing would like you to believe.

An op-ed about the failures of crime lab DNA testing at the New York Times -- written by Boise State Professor of Biology Greg Hampikian -- discusses this ongoing problem using some science of his own: a recently-released NIST study. (h/t Grits for Breakfast)

Researchers from the National Institute of Standards and Technology gave the same DNA mixture to about 105 American crime laboratories and three Canadian labs and asked them to compare it with DNA from three suspects from a mock bank robbery.

The first two suspects’ DNA was part of the mixture, and most labs correctly matched their DNA to the evidence. However, 74 labs wrongly said the sample included DNA evidence from the third suspect, an “innocent person” who should have been cleared of the hypothetical felony.

This is already a problem. People's lives are literally on the line and crime lab testing is more likely to make the wrong call on evidence than the correct one. What's truly disturbing is this study was completed in 2014, but the report was apparently buried by the scientists it implicated. As Dr. Hampikian states in his op-ed, the study's results may still be unpublished if it weren't for forensic scientists publicly complaining about the burial.

Four years have passed since the study's completion and it appears no improvements have been made. The study notes testing protocols vary widely and very little effort is being made to improve error-prone procedures. In addition, the study [PDF] comes with a disclaimer meant to dissuade litigants from challenging DNA evidence by quoting the study's findings.

The results described in this article provide only a brief snapshot of DNA mixture interpretation as practiced by participating laboratories in 2005 and 2013. Any overall performance assessment is limited to participating laboratories addressing specific questions with provided data based on their knowledge at the time. Given the adversarial nature of the legal system, and the possibility that some might attempt to misuse this article in legal arguments, we wish to emphasize that variation observed in DNA mixture interpretation cannot support any broad claims about “poor performance” across all laboratories involving all DNA mixtures examined in the past.

This certainly doesn't raise the reader's confidence in crime lab DNA testing. Instead, it gives the impression the four-year delay between completion and public release was for wagon-circling purposes as crime lab forensic scientists looked for ways to mute the impact of the study's findings.

But there are also problems with the study itself. The authors of the study appear far too willing to cut crime labs slack for their failures. Rather than point out the problems originating from a lack of standardized processes, the study uses them to excuse the failures, as if unintentionally nailing the wrong person for the crime was somehow worthy of gold stars for effort. Here's Hampikian's take:

It is uncomfortable to read the study’s authors praising labs for their careful work when they get things right, but offering sophomoric excuses for them when they get things wrong. Scientists in crime labs need clear feedback to change entrenched, error-prone methods, and they should be strongly encouraged to re-examine old cases where such methods were used.

The study confirms much of what has been exposed earlier: DNA evidence may be based on hard science, but any small variable -- including the inevitable tainting of DNA samples -- has the ability to throw things off. And when it's used as evidence in criminal trials, it has the potential to destroy lives. This study shows -- at least indirectly -- the labs handling DNA evidence aren't taking it nearly as seriously as they should.


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • identicon
    I.T. Guy, 27 Sep 2018 @ 4:25am

    "The first two suspects’ DNA was part of the mixture, and most labs correctly matched their DNA to the evidence. However, 74 labs wrongly said the sample included DNA evidence from the third suspect, an “innocent person” who should have been cleared of the hypothetical felony."

    "about 105 American crime laboratories and three Canadian labs"
    "most labs correctly matched their DNA"
    "74 labs wrongly said"

    108 - 74 = 30

    30 out of 108 got it right but
    "most labs correctly matched their DNA"
    Do they have a different definition of most?

    "However, 74 labs wrongly said the sample included DNA evidence from the third suspect"
    So they got it right and wrong?

    As far as the disclaimer if I were up on charges stemming from DNA evidence you bet you ass that I would bring this up. You're damn right I'd demand the samples be tested by multiple facilities and scrutinize the collection method and chain of custody.

    reply to this | link to this | view in chronology ]

    • identicon
      kallethen, 27 Sep 2018 @ 5:25am

      Re:

      Do they have a different definition of most?

      No, you misread. The "most labs correctly matched" part was specifically referencing the first two suspects who's DNA was included in the sample. Most labs matched them correctly.

      But the same can't be said for the third suspect. That person's DNA was not included in the sample, yet 74 said it was there.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 27 Sep 2018 @ 11:00am

        Re: Re:

        Step 1. Match everything to everything.
        Step 2. Boast of 100% success rate in finding matches.
        Step 3. Profit!

        reply to this | link to this | view in chronology ]

    • icon
      Bamboo Harvester (profile), 27 Sep 2018 @ 5:29am

      How soon they forget...

      It's only been what, about 20-25 years since the FBI lab was busted for just making up matches?

      108 labs, "most" (say 90) correctly matched two out of three in the blend. 74 of the labs incorrectly "found" the no-match control in the blend.

      This study was about *blended* samples, not *discrete* samples. Send them a cheek swab and they'll probably get it right. Send them a swab from a coin with 30 different DNA traces on it, they *might* get it right.

      In any event, ALL DNA evidence can be challenged and a sample of the evidence provided to the Defense for 3rd party testing. There is a caveat, where if a sample is too small for more than one test, the Judge MAY allow it. Or toss it out.

      reply to this | link to this | view in chronology ]

    • icon
      JoeCool (profile), 27 Sep 2018 @ 6:23am

      Re:

      If more than 70% (incorrectly) said there was a match across the board, it sounds like most labs are simply SAYING there's a match without running the tests. It clears the backlog, and they wouldn't be suspects if they weren't guilty, right?

      reply to this | link to this | view in chronology ]

    • identicon
      I.T. Guy, 27 Sep 2018 @ 7:15am

      Re:

      Thanks for clearing that up. :)

      reply to this | link to this | view in chronology ]

  • icon
    Coyne Tibbets (profile), 27 Sep 2018 @ 5:02am

    Not even a pretense of a fair test

    The study apparently wasn't even blind, which means the usual police/prosecutor pressure to match a suspect was lacking, and still 71% got it wrong.

    In real cases, with real pressure to get a conviction, it probably would have been 100%. Can't let the "bad guys" get away.

    reply to this | link to this | view in chronology ]

    • icon
      ShadowNinja (profile), 27 Sep 2018 @ 1:06pm

      Re: Not even a pretense of a fair test

      In real cases where the cops are certain it's someone who's already behind bars they'll fight tooth and nail to have no DNA testing, even fighting it beyond the statute of limitations, so that they couldn't prosecute the real criminal if it comes back with a different result.

      Prosecutors don't like DNA tests in 'sure win' cases because of the problem of DNA contamination. If two or more people's DNA are found it often opens up a pandora's box of things that Defense can raise to pin the blame on the mystery other person.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Sep 2018 @ 7:05am

    say it with me boys and girls

    man is not perfect

    reply to this | link to this | view in chronology ]

  • icon
    JohnnyRotten (profile), 27 Sep 2018 @ 7:12am

    Hmmmm.....

    DNA was scattered across several crime scenes before coming to the realization the DNA officers kept finding belonged to the person packaging the testing swabs used by investigators.

    Real life Dexter? Can't believe I'm the person that has to connect the dots here. :)

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Sep 2018 @ 7:21am

    District attorneys across the country have been aware of this for how long and yet continue to railroad innocent people into their private prison profit centers based upon flimsy to non-existent evidence.

    reply to this | link to this | view in chronology ]

    • icon
      Bamboo Harvester (profile), 27 Sep 2018 @ 7:40am

      Re:

      Don't blame the Prosecution. It's the Defense's job to challenge evidence.

      A retest at another lab that returns a different result is something the Prosecution does NOT want to see come up.

      There's a huge difference between enough evidence for an arrest and evidence to be presented at trial.

      DNA testing is still fairly expensive. Prosecutors don't care, it's tax money paying for it. So they'll routinely have DNA evidence retested before trial.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 27 Sep 2018 @ 7:57am

        Re: Re:

        "Don't blame the Prosecution. It's the Defense's job to challenge evidence"

        That is a rather lame excuse for the prosecuting attorneys who use dna evidence in this manner. I do blame the prosecution for bullshitting the court. Presenting false evidence could get one disbarred if judicial review boards were not corrupt.

        reply to this | link to this | view in chronology ]

        • icon
          Bamboo Harvester (profile), 27 Sep 2018 @ 8:05am

          Re: Re: Re:

          Again, Prosecutors routinely have DNA evidence RE-tested before trial.

          Two (or more) positive results may be why the Defense opts not to have a third test done.

          In the rare cases where DNA evidence *is* challenged, it's often because the sample was so small it was destroyed in the first test. When a Prosecutor is faced with that situation, they make sure the *rest* of their evidence is as solid as possible.

          There's no massive conspiracy here.

          reply to this | link to this | view in chronology ]

          • identicon
            Anonymous Coward, 27 Sep 2018 @ 9:10am

            Re: Re: Re: Re:

            "There's no massive conspiracy here."

            May not reach the level of conspiracy but it may clear the collusion hurdle.

            reply to this | link to this | view in chronology ]

          • identicon
            Canuck, 27 Sep 2018 @ 11:13am

            Math Fail

            "Again, Prosecutors routinely have DNA evidence RE-tested before trial."

            Duh. And 47% of the time, those two tests will both return false positives for an innocent party. And that's without any LE pressure applied.

            reply to this | link to this | view in chronology ]

          • icon
            nasch (profile), 27 Sep 2018 @ 4:18pm

            Re: Re: Re: Re:

            There's no massive conspiracy here.

            That doesn't mean there isn't a serious problem.

            reply to this | link to this | view in chronology ]

      • icon
        Anonymous Anonymous Coward (profile), 27 Sep 2018 @ 8:47am

        Re: Re:

        Which is more expensive, two or three more DNA tests at different labs, or incarcerating an innocent person for many years? That question is about money, then there is the moral question.

        reply to this | link to this | view in chronology ]

        • icon
          Bamboo Harvester (profile), 27 Sep 2018 @ 9:03am

          Re: Re: Re:

          You seem to be missing the point. DNA evidence may ALWAYS be retested at the request of the Defense, at a lab of the Defense's choosing.

          I gave the only exception to that - single sample destroyed by testing.

          reply to this | link to this | view in chronology ]

          • icon
            Anonymous Anonymous Coward (profile), 27 Sep 2018 @ 9:09am

            Re: Re: Re: Re:

            "DNA testing is still fairly expensive. Prosecutors don't care, it's tax money paying for it. So they'll routinely have DNA evidence retested before trial."

            Sorry, I was referring to this statement. It is one thing for the Prosecutor to retest using public money, it is another thing for a defendant to pony up for a test (just how expensive is it?). Either way, the Prosecutor has a responsibility to get it right.

            reply to this | link to this | view in chronology ]

            • icon
              Bamboo Harvester (profile), 27 Sep 2018 @ 9:22am

              Re: Re: Re: Re: Re:

              I was pointing out that the Prosecution routinely has DNA samples re-tested before going to trial. So the defense is fighting TWO positives from TWO labs.

              Which is why the Defense often doesn't request a third (or fifth) testing.

              Cost? Last I looked, 8-10 years back, a clean blood sample ran around $3,500 for a matchable stain.

              Frankly, the bulk of cases where DNA retesting is demanded by the Defense are either Public Defender or expensive private attorneys. In those cases, if it's NOT requested, there's usually a reason - if that third test ALSO matches, the Defense is sunk.

              reply to this | link to this | view in chronology ]

              • identicon
                Anonymous Coward, 27 Sep 2018 @ 12:46pm

                Re: Re: Re: Re: Re: Re:

                You seem to be implying that a conviction can be obtained upon dna evidence only, is there such a case?

                reply to this | link to this | view in chronology ]

          • identicon
            Anonymous Coward, 27 Sep 2018 @ 9:12am

            Re: Re: Re: Re:

            "DNA evidence may ALWAYS be retested at the request of the Defense,"

            That is why I read stories about BFE LEOs going after law students working to free innocent prisoners of the state.

            reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 27 Sep 2018 @ 9:17am

        Re: Re:

        >Don't blame the Prosecution. It's the Defense's job to challenge evidence.

        In the US, the defense rarely gets an opportunity, as prosecutors make a plea bargain the best option available to the accused, even if they are innocent.

        reply to this | link to this | view in chronology ]

        • icon
          Bamboo Harvester (profile), 27 Sep 2018 @ 9:25am

          Re: Re: Re:

          Prosecution presents it's case and evidence TO the Defense, including Discovery. THEN they give their "best" deal for a guilty plea.

          DNA testing and retesting is only done if it goes to Trial.

          For those wondering... Get a lawyer to watch ten minutes of any cop or "court" show. They'll spend the whole ten minutes yelling "OBJECTION!". It's ALL BS - NOT how the system actually works.

          reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 27 Sep 2018 @ 11:47am

        Re: Re:

        "It's the Defense's job to challenge evidence."

        Yes, the defense's job is to challenge the evidence, but that is extremely expensive. You have to pay for a attorney who is not overworked (aka not a public defender) and pay for a lab to perform the work.

        This creates a divide between those that have the resources and those that do not. We need a system where all persons are equal and the current system where the burden is on the defense to challenge bad science is not fair.

        We should hold prosecutors more accountable for junk/bad science used at trial.

        reply to this | link to this | view in chronology ]

  • icon
    lucidrenegade (profile), 27 Sep 2018 @ 7:42am

    But... CSI!

    reply to this | link to this | view in chronology ]

    • icon
      Bamboo Harvester (profile), 27 Sep 2018 @ 7:53am

      Re:

      The TV shows where lab geeks also collect evidence in the field, question suspects, carry guns, make arrests, and have $20,000,000 minimum in lab equipment?

      The show where everyone who has ever worked in a lab or overseen one yells at the TV over bad procedure if they have the ill luck to see a lab scene on TV?

      That CSI?
      :)

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 27 Sep 2018 @ 8:00am

        Re: Re:

        It is disgusting that tv shows continue to blow smoke up the collective asses of the general viewing public. Not just CSI, but there have been much worse shows that attempt to depict things that just are not real. No wonder a significant percentage of the populace believes really weird shit.

        reply to this | link to this | view in chronology ]

      • icon
        The Wanderer (profile), 28 Sep 2018 @ 5:57pm

        Re: Re:

        Those first several points fall under what is known as "conservation of characters", or a similar phrase; it isn't meant to indicate that things are actually like that, it's just a way to let the show present all the interesting parts without needing to either expect the audience to be familiar with a dizzying array of regular cast members (how many would be needed, to do everything shown as part of the investigation in a typical "CSI" episode?) or actually hire and pay enough people to fill all those roles on a regular basis.

        The bad-lab-procedure point is considerably less defensible, and I don't generally bother to try.

        reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Sep 2018 @ 8:16am

    Gold stars

    as if unintentionally nailing the wrong person for the crime was somehow worthy of gold stars for effort

    That's the way of the modern world: the worst possible products still get one gold star.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Sep 2018 @ 11:05am

    The customer is always right.

    Cop to "crime lab": Can you find us a match for this DNA?

    Crime Lab to cop: Sure. Who do you want it to match?

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 27 Sep 2018 @ 12:38pm

    DNA should have only been used to rule people out, not match suspects to crime scenes. Unfortunately, the people in crime labs are not much better informed than the people who end up as jurors.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 28 Sep 2018 @ 3:11am

    This is bad. Now it will be more desirable to shoot suspects and fugitives rather than attempt to prosecute using DNA evidence.

    reply to this | link to this | view in chronology ]

  • identicon
    Judy Burke, 28 Sep 2018 @ 9:11am

    Yes this is a major problem.

    reply to this | link to this | view in chronology ]

  • identicon
    Annonymouse, 28 Sep 2018 @ 9:49am

    DNA testing even without all the shenanigans by the whole chain from LEO to Prosecutors is only good for eliminating suspects.
    Always was.

    But but but theoretically it is one in a million... and that would give us how many viable suspects in the bay area alone?

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Close
Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Use markdown for basic formatting. HTML is no longer supported.
  Save me a cookie
Follow Techdirt
Techdirt Gear
Shop Now: Techdirt Logo Gear
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.