'Predictive Policing' Company Uses Bad Stats, Contractually-Obligated Shills To Tout Unproven 'Successes'

from the zoom-in-on-questionable-stats-and...-ENHANCE dept

We’re not quite at the stage where pre-crime divisions are being formed by forward-looking law enforcement agencies. Not yet. But we’re on our way.

SF Weekly has a very thorough report on PredPol, another entry in the “predictive policing” field. According to its website, PredPol’s software (“developed by a team of PhD mathematicians and social scientists” from a variety of California universities) algorithmically determines where crime is likely to occur and cranks out a map highlighting 500′ x 500′ “boxes” that are possible criminal hotspots — places officers should patrol more heavily.

PredPol is currently being deployed by a number of law enforcement agencies worldwide.

The company has sold its proprietary software here and abroad, from Kent County in England to Seattle, Wash., and here in the Bay Area to cities including Richmond, Los Gatos, Morgan Hill, and Santa Cruz.

San Francisco’s PD, despite having an early invitation to test drive the software, has yet to sign a contract with PredPol. Two things are preventing this from moving forward.

From the beginning, the effectiveness of PredPol had been a sticking point in the negotiations.

SFPD’s [CIO Susan] Merritt was skeptical. In a series of e-mails from July 2012 to August 2013, Fowler laid out the technical specifications for the software and the types of crimes PredPol claims to predict. “The crimes we predict are burglary [residential, commercial, auto], auto theft, theft, robbery, assault, battery, and drug crime,” Fowler wrote on July 23, 2012. “This goes significantly beyond your current … mapping tools,” he added. Affixed to [Donnie] Fowler’s e-mail signature was the claim that PredPol’s predictions are “twice as accurate as those made by vet cops…”

Merritt pressed Fowler about whether the program could handle violent crime. “Homicide is a priority in the department — and if it is not there it would just beg the question why not,” she wrote. Fowler admitted at the time that PredPol wasn’t predicting homicides and gun violence.

The skepticism is well-deserved. For one, PredPol simply hasn’t been around long enough to truly compare post-PredPol crime stats with pre-PredPol crime stats. It was still being tested in Santa Cruz and Los Angeles as of fall 2011, and has only been live in certain cities since the beginning of 2012. But that hasn’t stopped it from using mostly worthless year-to-year local crime data as “proof” of its crime fighting power.

A page on its site titled audaciously “Proven Results,” PredPol uses a couple of nearly context-free charts to tout its “success.”

This chart, titled “Proven Accuracy,” contains nothing to indicate what the divisions on the y axis represent, and while it touts its accuracy in the title, the sidebar says only this:

Predicted twice as much crime as experienced crime analysts in 6 month randomized controlled trials.

With nothing specified on the axis, PredPol could be simply outperforming analysts with a 20%-10% “accuracy.” And, as the lines proceed down the x axis, the 2-to-1 ratio shrinks, suggesting the more predictions PredPol makes, the less accurate it is. By the time the lines exit the chart, we’re looking at only a 9-6 [whatever] PredPol “advantage.” (Not only that, but the “2x” indicated is actually 3x, at least as far as I can tell from the unlabeled y axis.)

The second chart is nearly as bad.

Here it uses crime stats (oddly, “Crimes per Day”) from the Foothills Division of Los Angeles to claim a “13% reduction” in crime using year-to-year data. Looking at the chart shows that the $50,000 (and up) system reduced crime by one (1) crime per day (approximately) over that time period. Not only that, but the data only covers up to May 2012.

Crime stats published by the Foothills Division tell a different story. While PredPol claims this reduction is a success, additional statistics show the software has had a negligible effect on crime rates. At this point, according to PredPol’s own chart, the software has been running since late 2011, so any stats for 2012 and 2013 would be relevant. Some stats will have to be ignored as PredPol has only specialized in certain crimes since its beginning (“The crimes we predict are burglary [residential, commercial, auto], auto theft, theft, robbery, assault, battery, and drug crime…”)

Aggravated assault has seen a pretty steep drop, going from 310 in 2011 to 276 in 2012 and, finally, 242 year-to-date, an overall drop of 22% since PredPol’s addition to the force.

Other stats aren’t quite as cheery. Burglaries have dropped from 668 to 544 over that same time period (down 19%), but only viewing the two-year comparison ignores the fact that numbers have climbed since 2012, when only 494 burglaries occurred, a 10% increase over last year.

From 2011 to 2013, auto theft dropped all of 2%, from 712 to 700, a negligible difference. Unfortunately for PredPol, this is also a 10% increase over 2012 (634). Burglary/theft from motor vehicles has risen 5% over that time period (with a small dip in 2012). “Personal/other Theft” has declined 10% over two years and 16% in the last year.

While there are a couple of improvements, much of what’s here (increases and decreases) can be chalked up to normal statistical fluctuations. It’s not enough to completely rule out PredPol’s usefulness as a crime predictor, but it’s certainly not enough to be declared a “proven result” on its website.

Along with L.A.’s Foothill Division, Santa Cruz (where the pilot program took place) is also repeatedly highlighted on PredPol’s “Press” page. And once again, there’s nothing that conclusively shows PredPol’s worth as a predictive tool.

2012 to 2011 comparisons show an increase in criminal activity year-to-year (raw total: 697, up from 580 — a 20% jump), which would have occurred in the second year of PredPol’s deployment. There has been a drastic reduction in crime so far in 2013, but attributing that drop to PredPol also means attributing the 20% jump. A broader look at total crime statistics for the entire Santa Cruz area shows crime rates have fluctuated (sometimes greatly) with regularity over the past decade, suggesting past “solutions” have failed to produce predictable results. PredPol is presenting itself as just such a solution, but the data — even data pulled from the PDs it touts on its website — fails to bear that out.

As with any algorithm, more time in service means more data, and more data should mean better results. Optimistically, the recent decline in crime in Santa Cruz could be seen as an indicator that PredPol’s software is improving. But PredPol shows no such confidence in its ability to curb criminal activity. Sure, it presents itself on its public-facing website as an indispensable, cutting edge addition to any law enforcement agency’s set of crime-fighting tools, but the way it contractually obligates its customers to toot its horn for it (rather than wait for evidence of actual success) tells a completely different story.

PredPol has required police departments that sign on to refer the company to other law enforcement agencies, and to appear in flashy press conferences, endorsing the software as a crime-reducer — despite the fact that its effectiveness hasn’t yet been proven.

Turning customers into marketers via contractual obligations isn’t the act of a confident company. Even PredPol’s own marketing veers into some shady gray areas.

PredPol distributes news articles about predictive policing’s supposed success in L.A. to dozens of other police departments, implying that the company’s software has been purchased and deployed by the LAPD.

Statements from the LAPD say the opposite.

[I]t isn’t clear exactly whose software LAPD has been using. PredPol’s name does not appear anywhere in L.A.’s predictive policing records, though LAPD personnel say they are using PredPol’s software, and Malinowski’s contact information has appeared in PredPol’s sales literature distributed to other cities. In response to a public records request for contracts between L.A. and PredPol, the LAPD says no such agreements exist.

But this pitch, as shady as it is, still works.

PredPol gave the mayor and city council of Columbia, S.C. — Fowler’s hometown — a “confidential” briefing packet assembled by PredPol’s Brantingham. Inside were slides and graphs illustrating L.A.’s supposedly successful use of predictive policing to reduce crime.

This bit of marketing smoke netted two new contracts for PredPol — which led to even more obligatory PR work.

Swayed by the same claims, the city of Alhambra, just northeast of Los Angeles, purchased PredPol’s software in 2012 for $27,500. The contract between Alhambra and PredPol includes numerous obligations requiring Alhambra to carry out marketing and promotion on PredPol’s behalf. Alhambra’s police and public officials must “provide testimonials, as requested by PredPol,” and “provide referrals and facilitate introductions to other agencies who can utilize the PredPol tool.” And that’s just for starters.

Under the terms of the contract, Alhambra must also “host visitors from other agencies regarding PredPol,” and even “engage in joint/integrated marketing,” which PredPol then spells out in a detailed list of obligations that includes joint press conferences, training materials, web marketing, trade shows, conferences, and speaking engagements.

In addition to feeling used by PredPol’s marketing demands, agencies using PredPol’s software have also expressed concerns about how its reliance on “boxes” is diverting resources away from other needed areas — shifting too much focus onto wherever the algorithm decides today’s hot spots are. As much as some agencies may value the addition of predictive software to their arsenals, the worry remains that the “boxes” will receive an outsized amount of policing while the larger, unmarked areas go underserved.

While PredPol is nowhere near the point it can start targeting individuals before they commit crimes, the fact that a proprietary (and unproven) algorithm has the power (with the right marketing) to reroute police patrols is worrying, especially when no hard evidence has been presented that indicates there’s anything to PredPol’s claims. What’s happening here doesn’t necessarily infringe on anyone’s rights, but when combined with other questionable tactics (say, stop and frisk), it could result in pockets of abuse directly stemming from PredPol’s algorithmic “boxes.”

At best, it’s a marginally useful addition to the many tools PDs already deploy. But it’s certainly not the game changer PredPol’s presenting it as and its habits of presenting selective, limited data as proof and utilizing contracts that turn customers into unwilling shills isn’t exactly a confidence booster.

Filed Under: crime, police, predictive policing, shilling
Companies: predpol