In the previous chapter we treated the right to explanation as the tool that opens the black box. But once the box is opened, the next question is usually not technical. It is moral and legal: who is being harmed, and along what lines?
That is where algorithmic discrimination enters.
Artificial intelligence does not invent social inequality from nothing. It learns from data produced by institutions, policing patterns, labor markets, housing systems, and administrative records that already reflect human bias. When those patterns are fed into automated systems, the result is often not neutral efficiency but the industrialization of old disparities.
That is why the law has to do more than demand transparency. It has to ask whether a system produces unjustified unequal outcomes — especially in high-stakes settings such as criminal justice, immigration, employment, finance, and surveillance. On both sides of the Atlantic, the legal tools exist. They are not identical in structure, and the litigation paths they offer differ in important ways. But they aim at the same fundamental problem: preventing the state — or private actors exercising comparable power — from hiding discrimination behind code.
I. The United States: equal protection and the problem of intent
U.S. Constitution, Amendment XIV, § 1 (1868)
In the United States, the constitutional starting point is the Equal Protection Clause of the Fourteenth Amendment, which provides that no state shall deny to any person within its jurisdiction the equal protection of the laws.
That sounds broad. In practice, constitutional equality claims face a major structural limitation: disparate impact alone is usually not enough.
In Washington v. Davis, 426 U.S. 229 (1976), the Supreme Court held that a law or practice does not violate equal protection solely because it has a racially disproportionate effect. Justice White, writing for a 7-2 majority, held that constitutional claims generally require proof of discriminatory purpose — not only discriminatory outcome. Disproportionate impact is not irrelevant, but it is not the sole touchstone of an invidious racial discrimination forbidden by the Constitution.
That matters enormously for algorithmic systems. Most biased systems do not announce racial classifications openly. They rely on proxy variables: zip codes that correlate with race or ethnicity, prior arrest data shaped by decades of over-policing in minority communities, employment histories affected by structural inequality, language and location patterns that function as indirect markers of protected characteristics. The bias is built into the data. Nobody designed the system to discriminate. Washington v. Davis says that is not enough to establish a constitutional violation.
The constitutional argument is therefore more demanding than it first appears. The equal protection claim in an algorithmic context is rarely that the code explicitly classifies by race. It is that the government knows the system produces discriminatory effects and continues using it regardless. The stronger the evidence of known bias — and in the case of tools like COMPAS, that evidence is extensive and publicly documented — the stronger the argument that ongoing reliance constitutes something beyond neutral administration. When officials are aware that a system produces racially disparate outcomes and deploy it anyway in high-stakes contexts affecting liberty and detention, the question of whether that constitutes discriminatory purpose is no longer obviously answered against the plaintiff. Courts have been reluctant to make that finding, but the argument is available and the doctrine permits it.
II. Statutory law in the U.S.: where disparate impact has more force
Title VII of the Civil Rights Act of 1964, 42 U.S.C. § 2000e et seq. Fair Housing Act, 42 U.S.C. § 3604 et seq.
If the Constitution makes discriminatory intent hard to prove in algorithmic cases, federal civil rights statutes often provide a more tractable path — and the strategic choice between constitutional and statutory framing is one of the most important decisions in algorithmic discrimination litigation.
Under Title VII, the Supreme Court established the disparate impact doctrine in Griggs v. Duke Power Co., 401 U.S. 424 (1971), decided unanimously by Chief Justice Burger. The Court held that facially neutral employment practices may violate Title VII if they disproportionately exclude protected groups and are not sufficiently related to legitimate job requirements. Congress did not require discriminatory intent as a condition of Title VII liability. A practice that operates as a built-in headwind for minority groups is unlawful regardless of whether anyone intended that result.
The EEOC has made clear that the same principle applies directly to AI and automated employment tools. Its guidance confirms that federal law can be violated when an apparently neutral employment practice — including recruitment algorithms, hiring filters, screening tools, monitoring systems, and promotion criteria driven by machine learning — has an unjustifiable disparate impact based on a protected characteristic. That means algorithmic bias in employment is not a futuristic policy concern. It already fits inside an existing anti-discrimination framework, cognizable today in federal courts.
Outside employment, disparate-impact reasoning also survives in other statutory contexts. In Texas Department of Housing and Community Affairs v. Inclusive Communities Project, 576 U.S. 519 (2015), Justice Kennedy held for a 5-4 majority that disparate-impact claims are cognizable under the Fair Housing Act. Plaintiffs must identify a specific policy causing the disparity, but the intent requirement that forecloses many constitutional claims does not apply.
The picture is therefore more nuanced than it first appears. Constitutional equal protection is often intent-focused. Statutory discrimination law is often impact-sensitive. For the lawyer challenging algorithmic bias in employment, housing, or credit, the statutory path is frequently the more navigable route — and a well-constructed case combines both, using constitutional claims for damages and injunctive relief against government actors while pursuing statutory claims against the private sector and federally funded programs.
III. Europe: non-discrimination as a fundamental right
Charter of Fundamental Rights of the European Union — OJ C 326, 26.10.2012, Article 21 EU AI Act, Regulation (EU) 2024/1689, Article 10
The European Union starts from a structurally different position.
Article 21 of the EU Charter of Fundamental Rights prohibits discrimination on grounds including sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, disability, age, and sexual orientation. In the European legal order, non-discrimination is not merely a statutory policy objective. It is a fundamental right operating at the apex of the legal hierarchy we established in Chapter 7. That constitutional position changes the legal tone. The issue is not only whether a system creates a measurable disparity, but whether public and private actors are taking adequate steps to ensure that automated systems do not undermine a right protected at the highest level of European law.
The AI Act operationalizes that constitutional concern for high-risk AI systems. Article 10 requires that training, validation, and testing datasets be subject to appropriate data governance and management practices, including examination of possible biases and data gaps that could affect health, safety, and fundamental rights. Datasets must be relevant, sufficiently representative, and as free from errors and complete as possible in view of the intended purpose. This is not a guarantee that discrimination will never occur. But it is a legal admission of something structurally important: bias is foreseeable, and therefore governance must anticipate it. Organizations cannot claim they did not anticipate discriminatory outcomes from systems trained on historically biased data — Article 10 tells them to anticipate exactly that, and to govern against it before deployment.
In Europe, then, algorithmic discrimination is not only a downstream litigation issue. It is an upstream design and compliance issue. A high-risk AI system deployed without adequate bias testing and data governance is not merely a system that has not yet been challenged. It is already in breach.
IV. The reality behind the doctrine: COMPAS and facial recognition
Two examples have shaped the legal imagination around algorithmic discrimination more than almost any others.
The first is COMPAS. The ProPublica analysis, published in 2016, reported that the system was more likely to falsely flag Black defendants as high-risk than white defendants — labeling them wrongly at close to twice the rate for non-reoffenders. Whether one accepts every methodological choice in that analysis is not the essential legal point. The point is that COMPAS became the emblem of a structural phenomenon: a system presented as neutral, scientific, and objective can still distribute error unevenly across protected groups. And the group bearing the higher error rate in COMPAS was the same group that historically bore the higher burden of over-policing and harsher sentencing. The funhouse mirror reflects the shape it was given.
The second example is facial recognition. NIST’s ongoing evaluation program — the Face Recognition Vendor Test — has repeatedly documented demographic differentials in algorithm performance, including higher false positive rates for certain demographic groups in one-to-many search scenarios of the kind used in law enforcement identification. The individual harms those errors produce are not abstract: the Robert Williams and Porcha Woodruff cases, examined in Chapter 3, show what a false match means when the output drives an arrest.
Clearview AI crystallizes the European enforcement response to facial recognition at scale. The company built a database of more than 10 billion facial images scraped from the public internet without consent and sold facial recognition services primarily to law enforcement. European data protection authorities responded with a coordinated enforcement pattern. Italy’s Garante imposed a fine of €20 million in February 2022 for unlawful processing of biometric data, banning further collection and ordering erasure of all data held on Italian citizens. The Dutch Supervisory Authority imposed a fine of €30.5 million in May 2024 — the largest single GDPR penalty against the company — along with enforcement orders to cease ongoing violations.
These examples share the same structural pattern: historical or scraped data enters the system, the system produces outputs presented as objective, and the burden of error falls disproportionately on already vulnerable groups.
V. Why algorithmic discrimination is legally different from ordinary error
Ordinary administrative error is usually accidental and individualized. Algorithmic discrimination is more dangerous because it is repeatable, scalable, resistant to detection, and disguised as neutral optimization.
A biased human decision-maker may discriminate inconsistently. A biased algorithm reproduces the same pattern thousands or millions of times with apparent technical legitimacy, in ways that are difficult to detect without systematic audit. The mathematical veneer makes the discrimination harder to see and harder to challenge.
That is why equality law matters so much here. It changes the question from whether the system worked as designed to whether the system produced unequal treatment or unequal burdens that the law cannot justify. An algorithm that was designed correctly but trained on biased data is still producing discrimination — at scale, without conscious intent. The law does not require conscious intent to establish Title VII liability. In Europe, it does not require intent at all. That shift in framing is everything.
VI. Bias audits and rectification: from diagnosis to correction
You cannot correct algorithmic discrimination unless you first identify it. In practice, that means audits, impact assessments, and measurable outcome testing.
New York City’s Local Law 144 is important precisely because it institutionalizes bias auditing as a pre-deployment obligation for automated employment decision tools and requires public disclosure of the audit’s existence and certain results. That does not solve all discrimination problems. But it creates something that many AI systems otherwise lack: evidence. Once adverse impact is measurable and documented, the legal conversation changes. A plaintiff’s lawyer, a regulator, or a court can ask whether the model should be retrained on less biased data, whether the proxy variable should be removed from the input set, whether individuals affected by prior decisions are entitled to correction or review, and whether deployment should be suspended pending remediation.
In Europe, this logic connects directly to AI Act Article 10’s data governance obligations and to GDPR Article 16’s rectification rights. In the United States, it connects to Title VII, EEOC enforcement, administrative review, and — in criminal proceedings — the disclosure obligations that may require government actors to reveal the bias characteristics of tools used against defendants.
Bias audits, then, are not just compliance paperwork. They are the diagnostic mechanism that makes legal accountability for algorithmic discrimination possible — and the systems that resist audit, treating the measurement of their own discriminatory effects as a proprietary secret, are asserting that their opacity protects them from the legal consequences of their outputs. That assertion is increasingly difficult to sustain on either side of the Atlantic.
VII. Discrimination through code is still discrimination
One persistent temptation in AI governance discourse is to treat automated systems as if they belong to a separate legal universe. They do not.
If a credit score denies access because it encodes proxies for race, it raises the same equality concerns as any other discriminatory credit regime. If a detention-risk model systematically burdens one racial group more than another, the fact that the output was generated by software does not wash away the legal issue. If a facial-recognition system misidentifies minority individuals at higher rates, the resulting harm is not technical — it is legal and human.
The code may be new. The principle is not.
Discrimination through an algorithm is still discrimination. And the legal systems on both sides of the Atlantic were built, over decades of hard-fought litigation and legislation, to address that principle regardless of the mechanism through which the discrimination occurs.
Next: Chapter 15 — The national security exception. The black hole in both systems.
Leave a Reply