Chapter 2: The Blackbox Problem — The Legal Challenge of Our Time

In the previous chapter we built the vocabulary. We defined algorithms, machine learning, and the blackbox. We established something uncomfortable but unavoidable: opacity is not a defect in many AI systems. It is how they work.

Modern machine learning models operate through statistical correlations across enormous datasets. Their outputs can be measured. Their performance can be tested. But their internal reasoning often cannot be reconstructed in human-readable terms. For engineers, that is a technical challenge. For lawyers, it is something else entirely — because the legal system does not only care about results. It cares about reasons. Courts justify decisions. Administrative agencies must explain them. Defendants must be able to challenge them. The moment a decision affecting liberty, rights, or legal status is produced by a process that cannot be examined, the legal system encounters a structural tension. That tension is the blackbox problem.

When opacity becomes a legal problem

AI regulation begins with a simple observation: automated decision systems can affect fundamental rights. The EU Artificial Intelligence Act acknowledges this directly. It classifies systems used in law enforcement, migration, employment, creditworthiness, and access to essential services as high-risk — and once a system falls into that category, the law imposes a detailed governance architecture around it.

Five provisions are particularly important for lawyers. Article 9 requires providers to establish a continuous risk management system designed to identify and mitigate risks to health, safety, and fundamental rights throughout the lifecycle of the system — not just at deployment. Article 10 imposes data governance obligations: training datasets must be relevant, representative, and as free from errors and bias as reasonably possible. Article 12 mandates logging capabilities that ensure traceability of the system’s operation over time. Article 13 requires transparency regarding the system’s capabilities and limitations. And Article 14 requires deployers to ensure effective human oversight capable of preventing or minimizing the risks resulting from automated decision-making.

Taken together, these provisions reveal a fundamental regulatory insight. The law does not assume that algorithmic systems will become fully explainable. Instead, it attempts to build accountability around opacity. Where we cannot see inside the machine, the law demands records, oversight, and human responsibility above the output.

What a blackbox algorithm is in legal terms

Technically, a blackbox system is a model whose internal reasoning cannot be interpreted by human observers. In law, the concept is more precise. A blackbox algorithm is a decision-support system whose outputs influence legal outcomes while the reasoning process producing those outputs cannot be meaningfully examined by the individuals affected by the decision.

That matters because procedural justice relies on a basic guarantee: a person must be able to understand and challenge the reasons behind a decision that affects them. This principle appears across legal systems in different forms. In the European Union, it is reflected in the right to effective judicial remedy under Article 47 of the EU Charter of Fundamental Rights. In the United States, it appears through procedural due process under the Fifth and Fourteenth Amendments. When algorithmic reasoning becomes opaque, those guarantees become difficult — sometimes impossible — to exercise.

The problem becomes structurally more serious when the algorithm itself is protected by intellectual property law. And this is where the blackbox problem acquires its sharpest legal edge.

The conflict between trade secrets and due process

Many algorithmic systems used in high-stakes decisions are proprietary. In the European Union, the Trade Secrets Directive (Directive (EU) 2016/943) defines a trade secret as information that is not generally known, has commercial value because of its secrecy, and has been subject to reasonable steps to keep it confidential. In the United States, the Defend Trade Secrets Act (18 U.S.C. §1836) provides equivalent federal protection for confidential business information whose secrecy confers economic value.

Algorithmic models used in criminal justice, credit scoring, hiring, and immigration enforcement frequently fall under these protections. From a business perspective, that is entirely logical. Developing a predictive model requires significant investment. Protecting it as a trade secret is a legitimate legal strategy. The problem arises when that protection collides with the procedural guarantees of the justice system — when the evidence influencing a criminal sentence is the same information the defendant is legally prevented from examining.

Between 2013 and 2018, courts in no fewer than five states denied defendants access to algorithmic information on trade secrecy grounds. The pattern was consistent: a law enforcement agency uses a proprietary risk assessment tool, the developer invokes trade secret protection, and the defendant is left unable to examine the methodology behind the score that influenced their sentence, their detention, or their parole conditions.

The U.S. Congress has begun to respond. The Justice in Forensic Algorithms Act — introduced by Representatives Mark Takano and Dwight Evans — would prohibit trade secret privilege from being used to prevent defendants from accessing source code and other information about algorithms used to analyze evidence in criminal proceedings. The bill has not yet passed. But its introduction signals a recognition that the current balance is legally untenable. As Representative Takano stated directly: the trade secret privileges of software developers should never override the due process rights of defendants in the criminal justice system. The U.S. Department of Justice has itself acknowledged that the use of actuarial risk assessments in sentencing raises novel constitutional questions — a recognition that the conflict is real, documented, and unresolved.

State v. Loomis — when the blackbox enters the courtroom

In State v. Loomis, 881 N.W.2d 749 (Wis. 2016), the Wisconsin Supreme Court confronted this tension directly — and produced the case that now defines every discussion of algorithmic decision-making in criminal justice, AI bias in sentencing, and the right to challenge automated risk assessments in court.

Eric Loomis pleaded guilty to two charges arising from a 2013 drive-by shooting in Wisconsin in which he denied direct involvement: attempting to flee a police officer and operating a vehicle without the owner’s consent. At sentencing, data about Loomis was entered into COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — a proprietary risk assessment tool developed by Northpointe, now Equivant. The system analyzed Loomis’s questionnaire responses and criminal history to generate risk scores. It classified him as high risk of recidivism. The judge sentenced him to six years in prison.

Loomis did not challenge the conviction. He challenged the process. His argument was procedural and constitutional: the score influencing his sentence had been produced by an algorithm whose methodology was protected as a trade secret. He could not examine the model. He could not test its variables. He could not determine whether impermissible factors — including gender or socioeconomic proxies — had influenced the score. He could not, in any meaningful sense, challenge the tool that had labeled him dangerous.

During the appeal, the Wisconsin Supreme Court itself repeatedly questioned both parties’ counsel about how COMPAS actually works. As Justice Shirley Abrahamson noted in her concurring opinion, few answers were available. The court evaluating whether COMPAS violated due process could not fully explain how COMPAS worked. That detail alone captures the blackbox problem with precision.

The Wisconsin Supreme Court upheld the sentence — but with significant limitations. COMPAS could be used only as supplementary information, not as the determinative factor in sentencing. Judges must retain full discretion. And any presentence investigation report containing a COMPAS score must include a written advisory warning the judge that the system is proprietary and not fully transparent, that its predictions are based on group statistical data rather than individual analysis, and that the score cannot be the sole basis for sentencing.

In 2017 the U.S. Supreme Court denied certiorari, leaving the Wisconsin decision in place and the constitutional questions formally unanswered at the federal level.

What Loomis left unresolved — and why it matters for your practice

The Loomis decision is significant not for what it resolved but for what it exposed and declined to correct. The court acknowledged the ProPublica investigation documenting racial bias in COMPAS — then discounted it. It imposed warning requirements on judges — but left defendants without access to the methodology behind their scores. It accepted that due process could coexist with trade secret protection — but provided no mechanism for defendants to verify whether their score was based on accurate data or impermissible proxies.

The court held that a defendant could verify their COMPAS score was accurate by confirming they had correctly answered the questionnaire. That answer misses the point entirely. What defendants cannot verify — and what trade secret protection prevents them from verifying — is how the algorithm weighted the inputs, which factors it treated as predictive, and whether those factors functioned as proxies for race, gender, or socioeconomic status. We know what goes into COMPAS. We know what comes out. What we cannot know — and what the developer has consistently refused to disclose — is the process in between.

Not every court has reached the same conclusion as Wisconsin. In State v. Walls, a Kansas appellate court vacated the sentence and remanded for resentencing after finding that denying the defendant access to a complete risk assessment made it impossible to challenge the accuracy of information used to determine probation conditions. A 2017 federal trial court decision involving a teachers’ union in Houston held that a public agency’s reliance on a secret algorithm for high-stakes employment evaluations was incompatible with minimum due process requirements — reasoning that when a public agency makes high-stakes decisions based on secret algorithms, the proper remedy is to overturn the policy while leaving the trade secrets intact. The Federal Circuit has gone further still, holding that constitutional due process requirements can override statutory trade secret protections when the government relies on confidential information to make adverse decisions.

The trajectory is clear. The legal system is moving — unevenly but unmistakably — toward the position that due process rights must take precedence over trade secret protections when algorithmic tools are used in high-stakes government decisions. Loomis is the beginning of that journey, not its conclusion.

The EU approach — a different regulatory answer to the same problem

The European Union did not wait for courts to work through this conflict case by case. The AI Act addresses it structurally. For high-stakes systems — which explicitly include criminal justice and immigration applications — the combination of Articles 9, 12, 13, and 14 creates a transparency architecture that makes the Loomis scenario structurally more difficult to replicate in Europe. Mandatory logging means the record exists. Transparency obligations mean users must receive information about the system’s capabilities and limitations. Human oversight requirements mean a qualified person must be capable of understanding and intervening in the system’s outputs.

The GDPR reinforces this framework at the individual rights level. Article 22 establishes the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects — and the right to obtain an explanation of such decisions. This is not a constitutional principle developed through litigation. It is a positive legal right, enforceable before data protection authorities and courts across all EU member states.

This does not mean EU law has fully solved the blackbox problem. A system can log its inputs and outputs, maintain its documentation, and provide human oversight — and still be fundamentally opaque in its internal reasoning. What EU law has done is transform opacity into a compliance risk. A system that cannot be explained must still be governed. And non-compliance with the governance obligations creates liability.

For the transatlantic practitioner, the gap between these two approaches is operationally significant. A system lawfully deployed in the United States under the Loomis framework may not meet the transparency and oversight requirements of the EU AI Act. If your client operates in both jurisdictions — or if the system affecting your client was developed in one jurisdiction and deployed in another — you need to understand both frameworks and the distance between them.

The structural problem at the center of this book

The blackbox problem is not a criminal sentencing problem. It appears everywhere algorithmic systems influence decisions about people: predictive policing, immigration risk scoring, automated hiring, credit scoring, biometric identification, asylum processing. The sector changes. The architecture does not. Data about a person is collected. A statistical model processes that data. The model produces an output that influences a decision. When the reasoning behind that output cannot be meaningfully examined, the legal system faces a fundamental question: can a decision remain legitimate if the process that produced it cannot be scrutinized?

Every framework examined in this book — the GDPR, the EU AI Act, US constitutional due process doctrines, and governance standards such as NIST — represents an attempt to answer that question. Not by eliminating algorithmic systems. But by surrounding them with legal mechanisms designed to preserve accountability in the presence of opacity.

If you take one idea from this chapter, take this. AI governance is not primarily about technology. It is about legitimacy. Modern legal systems are built on a simple assumption: decisions that affect people’s rights must be reasoned, reviewable, and contestable. Blackbox algorithms challenge that assumption at its foundation. The laws you will study in the rest of this book are all attempts to preserve that legitimacy in a world where the reasoning process behind many decisions is no longer fully visible.

The blackbox problem is where that struggle begins.