The blackbox problem tells us that AI systems often cannot explain their decisions. But there is a second problem, equally serious and more counterintuitive: even when we can trace what a system did, the outcome may still be fundamentally unjust. Not because the machine malfunctioned. Because it worked exactly as designed — on data that was never neutral to begin with.
We tend to think of mathematics as objective. Replace a human judge — tired, biased, inconsistent — with an algorithm, and you get pure, neutral decisions. The technical reality has proven the opposite. Algorithms are not neutral. They are mirrors of our own history, and our history is not neutral either.
The reason is straightforward once you understand how machine learning works. A system trained on decades of judicial decisions does not learn justice. It learns statistics. It identifies the patterns that predicted outcomes in the past and applies them to new cases. If those historical patterns reflected systemic bias against certain groups — and in criminal justice, immigration, and lending, they did — the algorithm will encode that bias as a valid predictive rule. Discrimination becomes automated. And worse, it becomes hidden beneath a veneer of technological authority.
This is what ProPublica exposed in 2016, in its investigation into COMPAS — the risk-assessment software used across the United States to help judges determine sentences and parole decisions. The algorithm did not ask for race. It did not need to. It used proxy variables — neighborhood, family history, employment record — that are deeply tied to structural inequality. The result was documented and stark: Black defendants who did not go on to reoffend were flagged as high-risk nearly twice as often as white defendants in the same situation. White defendants who did reoffend were rated low-risk at significantly higher rates. The math was consistent. The outcome was discriminatory.
We will examine COMPAS in full in Chapter 25. For now, it serves as the clearest illustration of a principle that runs through every AI system operating in high-stakes environments: bias in, bias out, at scale, at speed, with the appearance of objectivity.
NIST, in Section 3.7 of the AI Risk Management Framework, gives lawyers the taxonomy they need to analyze this problem. It identifies three distinct categories of bias, each with different legal implications.
Systemic bias originates in the data itself — in the historical record of decisions made by institutions that were not always fair. No one programs this bias deliberately. It arrives with the training data and becomes embedded in the model’s predictions.
Computational bias emerges from how the model is built — from non-representative samples, from the choice of variables, from the metrics used to define what counts as a correct prediction. A model can be statistically accurate overall and still be systematically wrong for specific groups.
Human-cognitive bias enters through the people who design the system — who decide which data to use, which outcomes to optimize for, and how to measure success. These are judgment calls, and judgment is never neutral.
The legal significance of this taxonomy is that it tells you where to look. When your client has been harmed by an algorithmic decision, the question is not only whether the system was accurate in aggregate. It is whether it was fair to this person, in this context — and which layer of bias produced the outcome you are challenging.
NIST also makes a point that lawyers need to internalize: a system can be mathematically balanced and legally unfair at the same time. If a model produces equal error rates across demographic groups but those errors fall systematically on already disadvantaged populations, the formal equality conceals a substantive injustice. This is not a new legal concept. Since Aristotle, equity has meant correcting the application of general rules to account for human complexity. Bias mitigation in AI is the modern version of that same principle — the recognition that treating unequal situations equally is not justice.
For the practitioner, this means that the duty of care in AI cases now extends to the provenance of the training data. It is not enough to show that the system followed its instructions. You need to ask what those instructions were built on, whose history they encoded, and whose future they are now shaping.
Leave a comment