State Supreme Court Says Secret Software Used In Sentencing Determinations Not A Violation Of Due Process Rights

from the not-as-long-as-it's-used-perfectly-within-an-impossible-set-of-confines dept

An algorithm is deciding certain criminal defendants should spend more time in prison. And that determination can’t be fully challenged because the code belongs to a private company which provides the software to the government.

Eric Loomis was determined to be a “high risk” defendant, based on something called a “COMPAS score.” COMPAS — Criminal Offender Management Profiling for Alternative Sanctions — cranks out Presentence Investigation Reports for use in the courtroom, utilizing a number of factors to generate a score that lets judges know how likely the defendant is to re-offend.

The problems with this system are numerous. For one, the code is proprietary, so defendants aren’t allowed to examine the factors that lead to this determination, unlike other sentencing guidelines created by the government, which are open to the public to examine.

Another problem is that the algorithm engages in demographic profiling — generally considered to be a bad thing when it comes to determining criminal behavior.

Back in May ProPublica published an investigation into the risk-assessment software that found that the algorithms were racially biased. ProPublica looked at the scores given to white people and black people and then whether the predictions were correct (by looking at whether they actually committed or didn’t commit crimes); they found that in Broward County, Florida, which was using software from a company called Northpointe, black people were mislabeled with high scores and that white people were more likely to be mislabeled with low scores.

“Fits the profile” is the new “fits the description” — something that seems predisposed to putting blacks behind bars more frequently and for longer periods of time. Eric Loomis tried to challenge his COMPAS score but got nowhere with it, as the math behind it is locked up by Northpointe, which claims giving a defendant access to its trade secrets would pose a serious risk to its profitability.

Loomis argued that not giving him access posed a serious risk to his freedom. Allowing Northpointe to keep its algorithm secret was a violation of his due process rights, as it presented an unchallengeable score that could be used to keep him locked up longer than the normal range for the criminal activity he was convicted for.

His case went up the ladder to the Wisconsin Supreme Court, which has found [PDF] that defendants being unable to fully challenge a sentencing determination isn’t a Constitutional problem.

Ultimately, we conclude that if used properly, observing the limitations and cautions set forth herein, a circuit court’s consideration of a COMPAS risk assessment at sentencing does not violate a defendant’s right to due process.

We determine that because the circuit court explained that its consideration of the COMPAS risk scores was supported by other independent factors, its use was not determinative in deciding whether Loomis could be supervised safely and effectively in the community. Therefore, the circuit court did not erroneously exercise its discretion. We further conclude that the circuit court’s consideration of the read-in charges was not an erroneous exercise of discretion because it employed recognized legal standards.

Accordingly, we affirm the order of the circuit court denying Loomis’s motion for post-conviction relief requesting a resentencing hearing.

The downside of this decision is that Northpointe cannot be forced to hand over its algorithm for examination by criminal defendants. The upside is that the court has issues with using COMPAS scores to determine sentence lengths.

[T]he opinion comes with some interesting caveats about things judges need to keep in mind when using risk scores in sentencing decisions: The two most important factors they’re asked to keep in mind is that software has been found to be racially biased and that the software needs to be constantly monitored and updated with new information. (If you’re relying on data from five or ten years ago, it’s not going to be accurate.)

The court also notes in passing that the software was never intended to be used to determine sentence lengths. It was supposed to used by the Department of Corrections to assess risks posed by parolees or those requesting parole. But it does not go so far as to forbid the use of COMPAS scores in sentencing decisions. Nor does it suggest that opening up the algorithm for examination might bring much-needed transparency to the sentencing process. Instead, the Supreme Court says judges must walk a very fine line when utilizing COMPAS scores.

The queasiness that judges feel about algorithmic risk-assesment is reflected in the concurring opinion filed by Justice Patience Drake Roggensack. “Reliance would violate due process protections,” she writes. “Accordingly, I write to clarify our holding in the majority opinion: consideration of COMPAS is permissible; reliance on COMPAS for the sentence imposed is not permissible.

Unless a whole lot of judicial explanation accompanies every sentencing decision utilizing a COMPAS score, it’s going to be almost impossible for defendants to tell whether a judge has just “considered” Northpointe’s presentence investigation reports… or “relied” on them. Any sentence not hitting the upper end of the software’s recommendations could be viewed as mere “consideration,” even if “reliance” might be a more accurate term.

Without being allowed to closely examine COMPAS scores, defendants still aren’t being given a chance to challenge any erroneous information that might be included in these reports. The court’s reluctance to fully endorse the use of the software in sentencing decisions is a step forward, but it still allows judges to hand down sentences based on secret formulas that have already shown a predilection for recommending longer sentences to certain demographic groups.

Filed Under: , , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “State Supreme Court Says Secret Software Used In Sentencing Determinations Not A Violation Of Due Process Rights”

Subscribe: RSS Leave a comment
Mason Wheeler (profile) says:

The two most important factors they’re asked to keep in mind is … and that the software needs to be constantly monitored and updated with new information. (If you’re relying on data from five or ten years ago, it’s not going to be accurate.)

I have to wonder about that particular point. Are they really implying that they believe human nature is going to change that much in 5-10 years?

Anonymous Coward says:

Re: Re:

The data may involve things that do change in five or ten years. Some of this type of software accounts for what kind of neighborhood a defendant lives in. Neighborhood demographics and crime rates can change. Someone who lives in a neighborhood that gentrified in the last five years but was previously riddled with crime might get a harsher sentence because they are believed to live in a place where crime is likelier to occur.

Anonymous Coward says:

Re: Re: Re:

You seem to be very comfortable with it, is there any reason for this?

I would think the punishment for a crime should not be a variable dependent upon social standing, or any other unrelated data. But we have seen relaxed sentencing simply because the perp is from a rich family, and then said perp skipped home curfew for a vacation in Mexico.

Apparently family income is not a good indicator of recidivism, and yet it is used to predict whether the perp will be a problem in the future. Go figure.

Anonymous Coward says:

Re: Re: Re: Re:

Did you reply to the wrong post? Nothing I said implied I was comfortable with it. I just tried to offer an explanation why it would be necessary to have up-to-date data if you’re going to use this software for any purpose.

That speaks nothing to the fact that I don’t think this software should be used at all by judges. It removes the reason for having human judges in the first place. That said, human judges have been wrong (see the insufficient sentencing of Brock Turner or Ethan Couch) and need better oversight and accountability apparently.

Anonymous Coward says:

Re: Re:

The use of the software isn’t the real problem. The real problem is that we can’t look at what it’s doing. We can’t know on what factors the results were made.

I’d be perfectly happy leaving a lot of decisions in government up to a computer algorithm so long as that algorithm were fairly designed and we could examine it and see as much. It would be much preferable to semi-corrupt persons with their conflicts of interests and biases.

But this situation is disgusting. It’s a clear violation of not being able to examine the evidence and testimony against you.

Anonymous Coward says:

Re: Re: Re:

It’s a clear violation of not being able to examine the evidence and testimony against you.

While I agree with the general point of your post that this is a bad ruling and we should be able to inspect these algorithms, this is actually in a legal gray area. There is plenty of testimony during the sentencing phase that a defendant is not allowed to directly respond to, so it’s not without precedent (using the term colloquially here, not legally).

Anonymous Coward says:

Re: Re: Re: Re:

Democracy simply does not survive “gray areas” or “secrets” of any kind in law or its enforcement. To pretend that anything about law enforcement or the imprisonment of human beings should even be REMOTELY allowed to have any kind of secrets smacks of a corrupt and/or ignorant mind!

Richard (profile) says:

Re: Re: Re:2 Re:

Exactly – there used to be a phrase for that “Not only must Justice be done; it must also be seen to be done.”,_ex_p_McCarthy

Use of a secret computer program is in violation of that principle – and the linked case does actually look surpringly similar to the present one.

A quote from the decision on that case reads “Nothing is to be done which creates even a suspicion that there has been an improper interference with the course of justice. “

David says:

Re: Re:

Sorry, but computer or not, the whole “likely to become a repeat offender” thing makes sense with regard to rehabilitation programs, namely helping the defendant to regain footing.

It has nothing whatsoever to do with sentencing. You cannot punish people for things they haven’t done yet. Not if you pretend that this has anything to do with justice.

Secret sentencing criteria based on likelihood to reoffend will punish the same deeds harder for people who have a background making it harder to live honestly. So people who more deliberately commit crimes get let off the hook cheaper.

Wendy Cockcroft (user link) says:

Re: Re:

Here in the UK we have a TV show called Little Britain. One of the characters, customer service clerk Carol Beer, has a catchphrase, “Computer says no,” which is her stock response to queries from the public. This is the mentality we’re facing. The possibility of Terminator becoming a documentary is not at issue here, it’s “When are we going to stop hiding behind computers when we’re being jerks to each other?”

That One Guy (profile) says:

Can't have it both ways

If the software isn’t being used to influence the guidelines, and therefore can’t be challenged in court, then prohibit it from being used at all. Any use is going to influence the ruling, otherwise why even use it?

Conversely if the software is going to be used it absolutely should be open for examination and challenge by the defense, ‘profits’ be damned since we’re talking about people’s freedom here.

Anonymous Coward says:

Even if this software’s algorithms are revealed (and they likely will be; “secret evidence” etc.) the issue still raises some big unanswered questions.

Predicting whether a person is likely to (re)commit violent crimes runs into a serious problem that pits political correctness vs. cold hard reality. Police statistics show a huge (& highly predictable) correlation between a person’s race, sex, and age vs. his tendency to commit violent crimes.

If the goal is to protect individual liberty, then every person must be treated absolutely equally.

But if the goal is to protect society, then some rather Nazi-esque policies would be needed, such as “preemptively” jailing all inner-city black males between the ages of 15 and 30. There’s no question that crime would drop enormously, since that tiny demographic is responsible for a statistically huge amount of crime. The question is whether the concepts of equal-rights and due-process that form the cornerstone of modern Western society would allow such a thing, no matter how scientifically justified.

David says:

Re: Re:

The logic behind preemptive jailing would not make it possible to release the jailed persons ever again, so you’d better execute them once they cannot contribute to society any more in the form of labor.

See how much sense the Nazis made? Can we stop reviving their logic again and again and leave it in the graveyard of history?

Our justice system is supposed to punish people for what they did, not for who they are.

Uriel-238 (profile) says:

Re: Police statistics...

Are those police statistics based on the current high rate of convenient arrests and almost certain convictions?

Law enforcement has plenty of tools at their disposal by which to circumvent warrants to conduct unreasonable searches. Dog sniffs, for example can give a false positive up to 94% of the time. You just need the right dog.

We have more people in jail (per capita) than any other nation. We also have a solid likelihood that the majority of those people are falsely convicted. Plenty of them have major-crime sentences for petty crimes (e.g. possession).

And then there’s the matter that police file reports on what they want to file, and will omit filing on what they don’t care about.

So I wouldn’t trust police statistics to be an indicator of how how civilians behave, so much as how police behave.

Ninja (profile) says:

Re: Re: Police statistics...

That. And even if the statistics are right they don’t account for the time we’ve been systematically screwing those who are non-white since like forever (in the West at least). If you grow up being screwed and the only viable option to get some decent life and respect is crime it’s no wonder you’ll fall for it. And sadly even the black cops are racist sometimes.

Ninja (profile) says:

So the company can basically set up a random number generator behind the user interface and show the metaphoric middle finger to all the people being screwed by lady bad luck. Or a tyrannical Government could just set a number that fits their needs. Dissident? Set his score so he gets life sentence.

China approves. I wonder how long till they copy the process and actually manipulate the results.

Quiet Lurcker says:

Was Daubert decided in vain?

In the realm of computers and information technology, a program like this qualifies as an expert system – emphasis on expert, if you please – if only loosely. The software, and the people who purvey it claim that the software knows better than any human, which is the same claim made of nearly every bit of expert testimony, ever.

Since that is the case, shouldn’t the defense and courts be able to make an examination, either under Daubert, under Rule 702, or as a matter of common law?

As it stands, I’m with Anonymous Coward. Use of the software without outside review is a clear and direct violation of the Sixth Amendment.

JustMe (profile) says:

Sheesh - could the Court have punted any harder?

This is why jurors need to be aware of their rights. Just nullification is legal and proper, especially when one group of defendants is treated differently than other groups for the same crimes.

James Burkhardt (profile) says:

Re: Sheesh - could the Court have punted any harder?

Juries have the POWER to nullify. That comes as a natural extension of the design of our court system. what is debated is if they have the RIGHT to nullify. In several jurisdictions that will get you booted off the jury.

In either case, Jury Nullification has no baring on judges using software to decide sentencing lengths. You can not nullify sentencing, you’d have to nullify the conviction that happens before the sentencing, and we are talking about a problem that occurs even for those who should be punished.

orbitalinsertion (profile) says:

Never mind examining the algorithm, or how up to date the information is, but what information, how does it weigh, and how comprehensive is it?

Something like that could be hugely useful if done correctly. Not so much for sentencing unless except for mitigation. But “correctly” is something we fail to do quite often. (And then hide it, deny it, and make money from it. And further use it inappropriately on top of that.)

Wendy Cockcroft (user link) says:

Re: Re:

If you’ve ever wondered why…

Powell just wanted to entrench capitalism as the dominant ideology but right-wing activists took the memo and ran with it in directions he never envisaged. Result: activist judges, etc., pushing a fascist line in the courts and neoliberalism presented as a middle-ground position. I never realised how all-pervasive the infiltration was until recently. It’s all down to Powell, people.

Anonymous Coward says:

It appears to me to be more a violation of the 8th amendment than the 4th. That is, the courts have ruled that the 8th amendment prohibits punishments that are disproportionate to the crime. If you are incapable of explaining why a defendant received a certain punishment, then it is impossible to show that the punishment is in proportion to the crime.

Uriel-238 (profile) says:

Re: Disproportionate punishment

As much as I believe disproportionate punishment should be unconstitutional, I can see how the Eighth can be interpreted to at least not say that.

Mandatory minimums have done nothing but make for disproportionate punishments.

Of course the rule against cruel punishment should outlaw all the state and federal penal systems in the US. I’m not sure why it doesn’t. Not enough abuse?

Anonymous Coward says:

Even if you can inspect the algorithm, the one your looking at isn't neccessarily the one used.

Given variations in code due to software maintenance, or even virus or malware infection there is likely a number of problems related to code auditing that apply.

Not to mention the fat finger in the source data that seems to always show up after the warden gets a call from the governor, right?

So it isn’t just validation of the algorithm, but validation of the input data, which is I can almost guarantee will be badly tainted.

There will be socioeconomic factors that are reflected in the data that are not within the scope of the defendants control. That by itself, means the court is distinguishing formally based on factors that could be regarded as social class.

But I think that isn’t the point.

My guess is that COMPAS is mostly about externalizing liability. What they are doing is slopping a bunch of tech around, so that if they get called into court they can say “buh, buh, buh da COMPUTER said so!”. Which would be why it is important to insure that nobody knows what the computer is doing.

It is bad science. Reminiscent of eugenics in the early 1930s. They would do themselves a big favor by taking a hard look at this before it snowballs.

Uriel-238 (profile) says:

Re: The SCIENCE of eugenics worked.

Eugenics was just selective breeding. The problem is choosing which traits to favor and which traits to disfavor, which the Germans did on racial / mythical lines, and proved it’s too easy for the elites to bias.

But yes, if when they create a black box and give that box authority, it gives the appearance of displacing liability.

The problem is, of course, that people condemned by the black box are not going to accept that it is to blame, so much as the institution –and specific officer– that pushed it to the black box.

After the black box has incriminated some thousand or so people, I wonder if Florida judges will develop an attrition problem.

Willie Prevail says:

The real culprit stand

We are not all equal in the eyes of the law. This puts yet another black eye on the face of justice and of the trust one could surmise of those workings in the justice system.

Somehow, I still blame the bad guy committing multiple crimes knowingly who attempt to finagle the system who make it bad for everyone else when it might be their turn to seek justice.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...