ShotSpotter Employees Not Only Have The Power To Alter Gunshot Reports, But Do It Nearly 10% Of The Time
from the cooperating...-or-conspiring? dept
What’s being presented by ShotSpotter as good news for people who feel they’ve been wrongly accused, doesn’t actually appear to be all that comforting.
ShotSpotter’s mic tech and AI combine forces to report possible gunshots to law enforcement customers. It’s very hit or miss, he said with all possible puns intended. ShotSpotter says it’s nearly 100% accurate and can play an important part in reducing gun crime.
Actual customers say something else:
A 2013 investigation of the effectiveness of ShotSpotter in Newark, New Jersey revealed that from 2010 to 2013, the system’s sensors alerted police 3,632 times, but only led to 17 actual arrests. According to the investigation, 75% of the gunshot alerts were false alarms.
97% accuracy? Not what we’ve seen, says the San Diego PD:
A San Diego Police Department spokesperson told Voice of San Diego that during the four years ShotSpotter had been in use (as of September 2020) officers had made only two arrests responding to an alert and only one of those was directly linked to the alert.
Meanwhile, 72 of the 584 ShotSpotter alerts during that time period were determined to be unfounded, “a whopping 25 times higher than the 0.5 percent false positive rate put forth by the company,” the Voice of San Diego reported, based on data provided by the city’s police department.
Accuracy aside, can it help reduce gun violence and criminal acts linked to fired weapons? Again, the answer is no.
The City of Chicago Office of Inspector General’s (OIG) Public Safety section has issued a report on the Chicago Police Department’s (CPD) use of ShotSpotter acoustic gunshot detection technology and CPD’s response to ShotSpotter alert notifications. OIG concluded from its analysis that CPD responses to ShotSpotter alerts can seldom be shown to lead to investigatory stops which might have investigative value and rarely produce evidence of a gun-related crime.
That doesn’t mean the Chicago PD doesn’t think the tech is useful. In Chicago, officers still like ShotSpotter despite its inaccuracy because it allows them to do the sorts of things they want to do.
In reviewing ISR [investigative stop report] narratives for mentions of ShotSpotter alerts, OIG also identified 10 ISRs (13.9%) in which reporting officers referred to the aggregate results of the ShotSpotter system as informing their decision to initiate a stop or their course of action during the stop, even when they were not responding to a specific ShotSpotter alert. For example, some officers during the reporting period identified the fact of being in an area known to have frequent ShotSpotter alerts as an element of the reasonable suspicion required to justify the stop. Other officers reported conducting “protective pat downs” following a stop because they knew themselves to be in areas where ShotSpotter alerts were frequent.
A new Associated Press report — based on confidential ShotSpotter records shared with the news agency — is full of the sort of good news/bad news that tends to get saddled with noncommittal headlines, like this one: “Confidential document reveal key human role in gunshot tech.”
Here’s what’s notable in this report:
[A] confidential ShotSpotter document obtained by The Associated Press outlines something the company doesn’t always tout about its “precision policing system” — that human employees can quickly overrule and reverse the algorithm’s determinations, and are given broad discretion to decide if a sound is a gunshot, fireworks, thunder or something else.
Such reversals happen 10% of the time by a 2021 company account, which experts say could bring subjectivity into increasingly consequential decisions and conflict with one of the reasons AI is used in law-enforcement tools in the first place — to lessen the role of all-too-fallible humans.
The AP is technically correct. ShotSpotter says its tech can do what police can’t: be omnipresent with ears at the ready. What it pitches to cop shops is near perfection, a 97% success rate in hearing and locating gunshots.
What’s not made immediately clear is the human backstops. This is absolutely essential. Loud noises should not be instantly assumed to be gunshots. Hence the need for trained human employees to sort the “possibles” from the “confirmed.”
But there’s a downside to this — one that’s just as harmful as some PDs’ willingness to treat every suspected gunshot as blanket permission to violate the rights of those who happen to be in the reported vicinity. ShotSpotter’s human techs don’t just alter reports to distinguish things like a car’s backfiring from a suspected criminal’s gun firing. They also alter determinations and gunshot locations to better serve the needs of law enforcement agencies that interact with them.
On one hand, we have humans looking for AI errors. On the other hand, we have humans willing to cater to their law enforcement customers. A real land of contrasts sort of situation and one that doesn’t exactly inspire more trust in a cop tech company that has routinely overstated the accuracy of its main product.
Unsurprisingly, ShotSpotter execs remain bullish.
ShotSpotter said in a statement to the AP that the human role is a positive check on the algorithm and the “plain-language” document reflects the high standards of accuracy its reviewers must meet.
“Our data, based on the review of millions of incidents, proves that human review adds value, accuracy and consistency to a review process that our customers—and many gunshot victims—depend on,” said Tom Chittum, the company’s vice president of analytics and forensic services.
This is undoubtedly true. Human reviewers can make judgment calls the software can’t. This can help limit false positives. On the other hand, we’ve seen evidence that ShotSpotter’s human reviewers are not nearly as well-trained as the company claims. Their experts are not really experts. And, in at least two cases, the human reviewers have engaged in the sort of customer service that guarantees repeat government business (altering reports to better fit police narratives) but does little to protect the people who actually pay for these services: residents of cities where the tech has been deployed.
If ShotSpotter’s human staffers are altering gunshot reports 10% of the time, it means the software isn’t as accurate as the company claims it is. And it likely means they’re still altering reports by request for law enforcement agencies which may feel a false positive needs to be treated as an actual positive or feel the detection was too far away from their rights violations to be useful in their post hoc rationalization of their abuses.
Whatever the case, ShotSpotter should be treated with far more skepticism than what’s observed in this AP report. The company asserts facts not in evidence, sues journalists for truthfully reporting on its activities, and clearly considers itself to be an essential part of the criminal justice equation. Until the company is willing to let outside experts examine its tech, the company should be treated as part of the problem, rather than a cheap and easy solution to gun crime.
Filed Under: ai, altered reports, gunshot detection
Companies: shotspotter
Comments on “ShotSpotter Employees Not Only Have The Power To Alter Gunshot Reports, But Do It Nearly 10% Of The Time”
Probable cause...
Not for gunshots, but for bribery and manipulation by ShotSpotter.
Trust but verify...
If ShotSpotter is admitting 10% manual correction, you know that the real number is much higher. What company accurately self-reports their own failure rate without third-party confirmation?
This comment has been flagged by the community. Click here to show it.
It's probable cause, not evidence used in court
That really matters. The legal standard is not high. 90% accurate is WAY more than you need for probably cause. (heh, drug dogs). If it can get to 97% with human intervention.
The much larger issue is the legal regime of “everyone in the vicinity”, which in an urban area could be a hell of a lot of people. I think that’s a huge problem, personally, but it has nothing to do with the accuracy of the software, or accuracy with help, which in either case is fine.
Your real issue is the legal doctrine, not the software/service. Even if the humans were bending the service to help cops void the 4th amendment wink wink it would be nothing compared to what drug dogs do (and I suspect they’re not in the heat of the moment, mostly cuz the human operator has no idea what result the cop would “prefer” during the result of a typical response).
Re:
“Your real issue is..”
.. why there is violence (resulting in gunfire) in the first place.
The causes are many, and we as a society are ignoring most all of them. Rather than addressing the issues we grift off of the opportunities – this ends up in fuck you I got mine.
Re: Re:
DAFUQ? Make sense next time.
Re: Re: Re:
Nobody but you is responsible for your ongoing failure at literacy, matty.
Re:
“Your real issue is …” cost vs. value.
According to Newark police, the software led to arrests — and therefore had some value — .04% of the time.
According to San Diego police, the software has a PROVEN inaccuracy rate of AT LEAST 25%. That means 25% of the time, the software does nothing but waste police time that could be spent on something real.
This is not a useful use of taxpayer dollars that cops insist they require. Even if you think cops should be showered with money, a moment of critical thought would tell you that this is a shitty way to spend it.
Re: Re:
I don’t who you think is supposed to be the reliable source of data here. Presumably police departments are able to make cost/benefit decisions on allocated money. (Otherwise why give them any money at all?)
Re: Re: Re:
“..why give them any money at all?”
Many have been saying this for some time and based their track record, I have to agree.
What they have been doing is not law enforcement, it is law breaking .. and we are paying them to kill us. wtf
One more reason
Not to trust not just them,
But the surrounding industry
Fuck the dystopia
Re:
Its actually oblivion and uneducated people. Notice the abiguity on everything they say and do. Its a meaningless existence.
Dystopia is just digital slums made by people living in the slums anyways. Thats all they know.
Re: RE:
This^. The law enforcement / prison / industrial complex has become just as powerful and just as evil as the military / industrial complex. Really the only difference is that the law enforcement / prison / industrial complex is directing it’s evil, violence, and expense inward, against our own citizens, whereas the military / industrial complex directs it’s evil and violence outward, against people of other countries (except for the expense, of course, which still comes inward).
This garbage company is now in South Africa!
They are trying to push their rubbish tech on the government here for use in the Cape Flats. That is an area where there are a lot of gangs, thus a lot of shootings. Think somewhere like South Central LA.
There was a hugely unsuccessful trial that was very negatively received by the local residents. Shotspotter was completely inaccurate with loads of false positives, duh!
Unfortunately the party in charge of local government is the Democratic Alliance, which is the sole surviving white party from the Apartheid era! They are heavily pushing this Shotspotter technology.
The Cape Flats has always been a traditionally non-white area, so you can connect the dots…
1% is 1% too many. The sooner the cops use this to go after someone with the means to hire good lawyers who’ll rip this scam to shreds, the better.
ccording to the investigation, 75% of the gunshot alerts were false alarms.
…
Meanwhile, 72 of the 584 ShotSpotter alerts during that time period were determined to be unfounded, “a whopping 25 times higher than the 0.5 percent false positive rate put forth by the company,”
With numbers that bad ShotSpotter sounds rather like a digital version of police dogs, aka ‘probable-cause on four legs’ in that it’s primary purpose isn’t to find actual problems but give police legal cover for what they wanted to do, and any department looking to use it should be judged accordingly.
the human overrules the AI
IMO thats how it should be. The AI reports. The human validates. You should never take what an AI tells you as 100% accurate. Just so long as the employee isnt changing the report to comply with police requests I think its the right thing to do.
I would like to see the simulation routine they used to predict how the various different sound waves reverberate off buildings, the resulting interference patterns and how they then determine where the sound originates.
If they were really good at it, they could tell the dif between a pistol, a rifle, and a vehicle backfire.
So, it’s unreliable (in the direct of way too many false positives) and can be misused with little to no guardrails (as the results can be manually altered on demand).
Perfect tool for a police force that needs excuses for their abusive behavior rather than actual data.
Search everyone who is near the area of a crime? Sounds like a geofence warrant, just with most of the surveillance replaced with physical searches.