FBI's Facial Recognition Database Still Huge, Still Inaccurate, And DOJ Shows Zero Interest In Improving It


The FBI’s biometric database continues to grow. Its Next Generation Identification system (NGI) is grabbing everything it can from multiple sources, compiling millions of records containing faces, tattoos, fingerprints, etc. from a blend of criminal and non-criminal databases. It went live in 2014, but without being accompanied by the Privacy Impact Assessment (PIA) it promised to deliver back in 2012.

Lawsuits and pressure from legislators finally forced the FBI to comply with government requirements. That doesn’t mean the FBI has fully complied, not even two years past the rollout. And it has no interest in doing so in the future. It’s currently fighting to have its massive database exempted from federal privacy laws.

Much of the information we have about the FBI’s NGI database has come from outside sources. The EFF and EPIC have forced documentation out of the agency’s hands via FOIA lawsuits. And now, the Government Accountability Office (in an investigation prompted by Sen. Al Franken) is turning over more information to the public with its review of the system.

What the GAO found is more bad news. The FBI is all about collecting data. It has little interest in ensuring the data is accurate or even usable.

The FBI’s system searches not just its own database, but also photo databases maintained by seven participating states, the US Department of State – which issues passports – and the US Department of Defense, shared among federal law enforcement agencies and the participating agencies, though access on the state level is obtained through the FBI.

This is only part of the NGI. To amass the 411 million photos it has collected to this point, the FBI dumps in the contents of a national criminal database.

[T]he GAO report found a much larger program, run by the criminal justice information services division of the FBI (CJIS), called Facial Analysis, Comparison and Evaluation, or Face, which “conducts face recognition searches on NGI-IPS and can access external partners’ face recognition systems to support FBI active investigations”.

The multiple inputs — which allow criminal and non-criminal biometric data to intermingle — still return an alarmingly high number of false positives. According to data obtained by EPIC, the facial recognition portion shows an error rate of 15-20% in the top 50 results returned from searches. That was the error rate in 2010. We can assume the hit rate has improved since then, but we have no way of knowing what the current error rate is because the FBI is uninterested in policing the accuracy of its database.

From the GAO report [PDF]:

Prior to deploying NGI-IPS, the FBI conducted limited testing to evaluate whether face recognition searches returned matches to persons in the database (the detection rate) within a candidate list of 50, but has not assessed how often errors occur. FBI officials stated that they do not know, and have not tested, the detection rate for candidate list sizes smaller than 50, which users sometimes request from the FBI… Additionally, the FBI has not taken steps to determine whether the face recognition systems used by external partners, such as states and federal agencies, are sufficiently accurate for use by FACE Services to support FBI investigations

The GAO report also points out the FBI has been severely delinquent in its obligations to the public. Reports it was supposed to deliver prior to rollout have only just recently appeared, including one release apparently prompted by the GAO’s assessment of the NGI program.

NGI-IPS has been in place since 2011, but DOJ did not publish a System of Records Notice (SORN) that addresses the FBI’s use of face recognition capabilities, as required by law, until May 5, 2016, after completion of GAO’s review. The timely publishing of a SORN would improve the public’s understanding of how NGI uses and protects personal information.

The GAO has made six recommendations to the agency, three of which are being disputed by the DOJ. According to the DOJ, the reason for the mandatory reports being delivered after-the-fact doesn’t need to be examined because the FBI “has established practices that protect privacy and civil liberties beyond the requirements of the law.” This sounds like the FBI has “nothing to hide,” which is at odds with the lack of responsiveness by the agency to demands for updated PIAs and SORNs over the last eight years.

The DOJ also disagrees that it should have to audit the facial recognition database’s “hit rate,” something that was only 80-85% accurate five years ago. (In fact, the FBI’s specifications consider 85% accuracy to be acceptable when returning lists of possible suspects.) The DOJ claims the database can never return a false positive because it apparently has enough manpower and resources to chase down every bogus lead.

In its response, DOJ stated that because searches of NGI-IPS produce a gallery of likely candidates to be used as investigative leads instead of for positive identification, NGI-IPS cannot produce false positives and there is no false positive rate for the system.

The GAO understandably disagrees. Accuracy is important, especially if the FBI is going to put innocent people under investigation… or overlook potentially dangerous suspects.

Without actual assessments of the results from its state and federal partners, the FBI is making decisions to enter into agreements based on assumptions that the search results may provide valuable investigative leads. In addition, we disagree with DOJ’s assertion that manual review of automated search results is sufficient. Even with a manual review process, the FBI could miss investigative leads if a partner does not have a sufficiently accurate system.

The DOJ apparently still feels a 20% chance of putting the wrong person under investigation is still acceptable. And it still believes that it’s so far ahead of the privacy curve that it doesn’t need to apprise the public of the potential privacy implications of its massive biometric database. The information forced out of its hands by litigants and outside agencies shows the FBI is far more interested in collection than dissemination — that it should be able to take all it wants from the public without having to hand out anything in return.

Filed Under: , , , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “FBI's Facial Recognition Database Still Huge, Still Inaccurate, And DOJ Shows Zero Interest In Improving It”

Subscribe: RSS Leave a comment
Anonymous Coward says:

Investigating the wrong people is fine

As the FBI demonstrated with the Orlando shooter, investigations go nowhere, even when they are aimed at the right people. Therefore, there is no harm in investigating the wrong people, or even just investigating random people with no connection to crime. Besides, the FBI is far too competent and trustworthy for a misplaced investigation to inconvenience the innocent subject.


Anonymous Coward says:

When you label every american citizen a potential domestic terrorist then that’s a feature not a bug. America is a tyranny that has created a facist police state for itself. Real Shame though.

Be curious if people will roll over and accept it when they finally remove the 2nd amendment that keeps getting in their way of total control. by that I mean they cannot control an armed population through fear and terror tactics.

Wendy Cockcroft (user link) says:

Re: Re: Re:

And it will continue to do so as long as the “important” rights remain. For all the talk the only action has been from a bunch of fringe loons, who, due to the fringe loons on the right, are allowed to run amuck until the shooting starts, in which case one gets killed and the other treated as a terrorist, as predicted by yours truly.

No armed rebellion could ever succeed in America; a) you couldn’t get enough people to join in and b) you’d be outgunned, outnumbered, and out on every media platform as an evil terrorist and threat to national security, etc.

You’re better off working towards a peaceful solution via the democratic system.

That One Guy (profile) says:

False positives are a feature, not a bug

When all you care about is justification for a search or investigation that you might otherwise not be allowed to perform the more ‘hits’ the better(see: drug dogs). Accuracy only matters if you care whether or not your actions impact an innocent person, and I think it’s fairly safe to say that very few government agencies care about avoiding that these days.

If a system had a 60% accuracy rate for example it would return hits on an innocent person four out of every ten times.

To someone that prioritizes protecting the innocent over finding/punishing the guilty that’s four people searched/investigated that shouldn’t have been.

For someone who prioritizes finding/punishing the guilty over protecting the innocent on the other hand that’s four searches/investigations that they otherwise wouldn’t have been able to do, four more ‘chances to find a criminal’.

That Anonymous Coward (profile) says:

Movies make shity policy...

They saw a movie where facial recognition worked in milliseconds and always found the bad guy, so they decided it could be built. I mean technology in movies is totally based on reality.

This isn’t the first time we’ve seen this sort of magical thinking burn us. Billions of dollars wasted on programs that just don’t work in reality, as those championing them are sure if they did it in the movies we can do it IRL.

Where is the push back for these failed bad movie plot elements? Why do they keep spending money on magical things that those selling can’t actually deliver what they promised. Why don’t they demand examples run by outside firms rather than computer renderings of how they imagine it could work in 20 years of unlimited spending?

Udom (profile) says:

Facial recognition

Facial recognition systems are interesting because they are still so inefficient despite all the money and effort thrown into them. Which is more dangerous, a surveillance system that is highly inaccurate, or one that doesn’t make mistakes? Crows have been demonstrated to be able to reliably recognise a human even if he/she is disguised.

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...