We have established that algorithms discriminate because they learn from biased history. But there is a deeper problem underneath that one, and it is the problem that will follow you into every courtroom, every administrative appeal, and every client consultation in this field: even when an AI system is working exactly as intended, nobody — not the developer, not the deployer, not the regulator — can fully explain why it reached a specific conclusion.
This is not a bug. It is an architectural feature of the most powerful AI systems in use today.
At the heart of every legal system lies an obligation that we rarely question because it seems self-evident: decisions must be reasoned. A judge does not merely decide — they explain why, with reference to law and fact, in terms that the losing party can read, challenge, and appeal. This duty to give reasons is not a procedural formality. It is the mechanism by which the rule of law maintains its legitimacy.
AI breaks this mechanism. Not always, and not irreparably — but in ways that the law is only beginning to confront.
The European legislature acknowledged this directly. Recital 47 of the AI Act admits what engineers have known for years: certain AI systems are so complex that their functioning is inscrutable even to their own creators. The mathematics involved — millions or billions of numerical weights interacting across multiple layers of a neural network — do not produce an argument. They produce a result. And the path from input to output cannot be reconstructed in human-readable terms.
The legal consequence the AI Act draws from this is precise. If a citizen cannot understand why an algorithm denied them parole, flagged their asylum application, or placed them on a deportation list, they cannot effectively exercise their right to appeal. Opacity is not merely a technical inconvenience. It is a direct threat to fundamental rights.
NIST, in Section 3.5 of the AI Risk Management Framework, provides the analytical tools to take this problem apart. It distinguishes explainability from interpretability — two concepts that sound similar but operate at different levels of the legal analysis.
Explainability describes the system’s ability to represent its own internal functioning in terms a human can follow. It answers the question of how — what mechanism connected this input to this output. Interpretability goes further. It answers the question of why this particular result matters in the context of this particular decision. Not how the model works in general, but why this person, in this case, received this outcome.
To see why the distinction matters, consider a simple analogy. If a mechanical engine fails, a specialist can disassemble it, identify the broken component, and trace a linear chain of physical causation from fault to failure. Traditional legal reasoning works similarly — a judge builds a logical bridge between rule and fact, and any competent lawyer can audit that bridge argument by argument. Deep learning models operate on a different principle entirely. They do not follow logical rules. They operate through trillions of numerical correlations that no human mind can follow in sequence. There is no component to point to. There is no argument to read. The system produces correct results — or incorrect ones — for reasons that exceed human intuition.
NIST is clear about the consequences: without explainability, there is no accountability. A system that cannot be audited cannot be governed. And a system that cannot be governed has no place making decisions about liberty.
The legal frameworks respond to this with two categories of safeguard that every AI lawyer needs to understand.
The first is mandatory human oversight. The AI Act requires that high-risk systems be designed to allow human intervention — not as an afterthought, but as a structural element of how the system operates. In practice, this takes two forms. Human-in-the-loop means a person intervenes at every individual decision: the AI proposes, the human validates. Human-on-the-loop means a person supervises the system’s overall operation and retains the authority to override or shut it down when anomalous behavior is detected. The difference matters for liability. A human-in-the-loop configuration distributes responsibility across each decision. A human-on-the-loop configuration concentrates it in the oversight function — and raises hard questions about what a supervisor is actually required to notice and when.
The second safeguard is documentary. The AI Act imposes on providers of high-risk systems an obligation to maintain technical documentation and automatically generated event logs for ten years. This is the flight recorder principle applied to software. We may not understand today why a specific decision was made. But the obligation to preserve the record ensures that a forensic audit remains possible tomorrow — that accountability does not expire simply because the moment of decision has passed.
Together, these two frameworks — the AI Act and NIST — arrive at the same conclusion from different directions. Efficiency cannot stand above dignity. A decision that cannot be explained is not an intelligent decision. It is, in legal terms, a decision that was never properly made.
That is the problem this book is built around. Everything that follows — the bias audits, the right to explanation, the constitutional challenges, the discovery motions — flows from this single inescapable fact: we are allowing systems nobody can fully explain to make decisions that determine whether people go to prison, lose their homes, or are deported from the country where they live.
The law has tools to respond. This book is about learning to use them.
Leave a comment