Computer Says No: Algorithmic Discrimination

Computer Says No — and Won’t Say Why

If you watched British television in the early 2000s, you probably remember Carol Beer. She was the bank clerk, the travel agent, the hospital receptionist, always the same character, always the same expression: blank, mild, faintly pleased with herself. She would tap her keyboard with two fingers, study the screen, and deliver her verdict without inflection. Nobody called it algorithmic discrimination then. It was too absurd to need a name. But the shape of it, a system making a decision about a person, with no explanation offered and no appeal accepted, was already precisely right.

Computer says no :(

No explanation. No appeal. No suggestion that the matter might be reviewed or the decision reconsidered. The computer had spoken. Carol had relayed the message. Transaction complete.

The sketch was funny because it recognised something real, the growing absurdity of bureaucratic systems that had outsourced their decisions to machines and their accountability to nobody and inflated it just enough to be absurd. You laughed because you’d been Carol Beer’d, by a bank or a government department or an insurance company, and the recognition was cathartic.

It is less funny now when you really think about it in the context of our contemporary AI world. Not because the writers lost their touch, but because the gap between the sketch and reality has closed. The algorithms making consequential decisions about your mortgage or your job application, are offering exactly as much explanation as Carol Beer. And unlike Carol, they are not performing indifference. They genuinely cannot tell you why.

What These Systems Actually Are

Before the horror, the mechanics — because understanding what these systems are is part of understanding why they fail the way they do.

Algorithmic hiring and credit scoring systems are built on the same basic principle. You feed a system historical data: past hiring decisions, past loan outcomes, past credit repayments, and you ask it to find patterns. Which applicants became successful employees? Which borrowers repaid their loans? The system finds the characteristics that correlate with those outcomes and uses them to score new applicants.

This sounds reasonable. It is, in a narrow technical sense, exactly what it claims to be: pattern recognition applied to large datasets. The problem is not in the mechanism. It is in what the mechanism is trained on.

Historical hiring data in most industries reflects decades of human bias. The people who were hired and defined as “successful” were disproportionately white, disproportionately male, disproportionately from certain educational backgrounds and postcodes and social networks. When you train a system on that data, you are not training it to find the best candidates. You are training it to find candidates who resemble the people who were hired before, who were hired, in many cases, because of conscious or unconscious preferences that had nothing to do with the job.

Amazon discovered this the hard way. In 2018 they scrapped an AI recruitment tool they had been developing for years, after internal testing revealed it was systematically downgrading CVs that included the word “women’s” as in “women’s chess club” or “women’s college.” The system had been trained on CVs submitted to Amazon over a decade, in a male-dominated industry. It had learned, faithfully and accurately, that the company’s previous successful hires had not submitted CVs with that word in them. It was doing exactly what it had been told. What it had been told was, it turned out, the problem.

Computer Says No — and Won’t Say Why

The Amazon story has a reasonably satisfying ending: the system was caught, scrapped, and the decision to abandon it was made public. Most of the time this does not happen, because the defining characteristic of these systems is not just that they make decisions, it is that they make decisions nobody can fully explain.

This is what researchers and lawyers call the black box problem. The system takes inputs, produces outputs, and the relationship between the two is, in many cases, opaque even to the people who built it. Machine learning models, the kind used in sophisticated credit scoring and hiring systems, do not operate on explicit rules that can be listed and examined. They operate on patterns in data, weighted in ways that produce accurate predictions without producing comprehensible explanations.

The consequence, for the person on the receiving end, is Carol Beer at industrial scale.

UK consumers applying for credit have, in most cases, no legal entitlement to a meaningful explanation of why their application was refused. They can be told that a credit reference agency’s data was used, and they can access that data, but the algorithmic processing that turned that data into a decision, the actual reasoning, remains the commercial property of the institution, protected behind opacity that the law has not adequately penetrated. A 2025 academic analysis published in Computer Law & Security Review argued for a statutory right to explanation in automated credit decisions, noting that “algorithms are unpredictable and can make unreliable decisions” and that the opacity problem, the black box, makes the informed consent process in automated credit decision-making fundamentally incomplete.

What makes this particularly striking is the timing. In early 2025, the Court of Justice of the European Union confirmed, in the Dun & Bradstreet Austria case, that EU citizens are entitled to a genuine explanation of the logic and results of automated decisions affecting them. The EU was strengthening the right to explanation. At almost exactly the same moment, the UK’s Data (Use and Access) Act 2025, which received Royal Assent in June 2025, repealed the UK GDPR’s Article 22 protections and replaced them with a more permissive framework, moving in the opposite direction, weakening the protections that existed just as their value was being confirmed in European law.

The irony is not subtle. Post-Brexit Britain has loosened the rules governing algorithmic decisions about its citizens at the precise moment the EU has tightened them.

The Cases That Make It Concrete

Abstract arguments about black boxes and regulatory frameworks have a way of floating above the actual experience. The cases ground it.

The deaf applicant – In March 2025, the ACLU Colorado filed a complaint against Intuit and its AI vendor HireVue. An Indigenous and deaf job applicant had applied for a position at Intuit. HireVue’s video interview platform, which analyses facial expressions, speech patterns, and tone of voice to score candidates, assessed the application and provided feedback. The feedback included a recommendation to work on “active listening.” The system had assessed a deaf person on audio cues and concluded they needed to improve their listening skills. Nobody caught it. The computer said no and explained why, and the explanation was an absurdity that a human reviewer would have caught in seconds.

The neurodiverse candidates – Research published in ScienceDirect in late 2025 documented that HireVue and similar platforms disproportionately disadvantaged neurodiverse applicants, people with autism or ADHD, by scoring them lower due to non-traditional response patterns. Eye contact, speech rhythm, response speed: the system had been trained on neurotypical norms and flagged deviation from those norms as a negative signal. The candidates weren’t performing poorly at the job requirements. They were performing differently from what the algorithm expected.

The name problem – Research published in 2025 found that LLM-based resume screening disadvantaged applicants with Black and female-associated names, even when all other resume content was identical. The algorithm was not programmed to discriminate by name. It had learned to associate certain name patterns with outcomes that correlated, in its training data, with less successful hire histories. The discrimination was emergent , nobody designed it, everybody produced it.

The monoculture problem – A Stanford HAI study published in May 2026, the first large-scale analysis of hiring algorithms in the wild, found that 90% of US employers use AI screening tools and most rely on the same few third-party vendors. The implication is significant: if a bias exists in one widely used system, it does not affect one employer’s hiring pool. It affects millions of applications simultaneously, across hundreds of companies, with no human reviewer catching the pattern because no single employer sees enough of the picture to recognise it.

These are not edge cases or theoretical concerns. They are documented, current, and scaling.

Who Bears the Cost

The costs of these systems are not distributed evenly. They fall, with dispressing consistency, on the people the systems were least designed to serve.

The candidates whose names trigger lower scores. The applicants whose accents deviate from the training data’s norm. The neurodiverse people whose communication styles don’t fit the algorithm’s model of a successful interview. The people in postcodes that correlate, through decades of structural inequality , with higher credit risk, regardless of their individual financial behaviour. The people who have always been most likely to face discrimination in hiring and lending are now facing that discrimination automated, accelerated, and made opaque.

This is the central point this blog, and the broader argument in Well… How Did We Get Here?, keeps returning to: these systems do not intend to discriminate. They do not intend anything. Intention is a human capacity these systems don’t possess. But they produce discriminatory outcomes because they were trained on discriminatory data, designed by people who did not adequately interrogate their assumptions, and deployed at scale before anyone fully understood what they were doing.

The technology shaped the outcome without anyone deciding to shape it that way. That is the definition of technological determinism operating in the wild, and it is precisely why the question of who builds these systems, on what data, under what oversight, matters so much.

What Is Being Done and What Isn’t

The regulatory landscape is moving, albeit unevenly and not always in the right direction.

New York City now requires annual independent bias audits for automated employment decision tools, with public reporting of results, before those tools can be deployed. California finalised regulations in October 2025 clarifying how existing anti-discrimination law applies to AI hiring tools. The Colorado AI Act, effective June 2026, requires developers and users of AI hiring tools to use reasonable care to prevent algorithmic discrimination.

These are meaningful steps. They are also partial, fragmented, and jurisdiction-specific, which means that a company operating across multiple territories faces a patchwork of requirements, and a job applicant’s rights depend heavily on which postcode their application is processed in.

The EU AI Act categorises high-risk AI systems, including those used in employment and credit decisions, and imposes transparency and accountability requirements that will shape how these systems are built and deployed across Europe. Portugal, as an EU member state, is within that regulatory perimeter.

The UK is not. And the Data (Use and Access) Act 2025’s weakening of automated decision protections means that UK consumers are, at this specific moment, less protected than their EU counterparts against algorithmic decisions that affect their financial and working lives. Whether that changes depends on political will and public pressure that is, right now, significantly less organised than the commercial interests of the companies that build and sell these systems.

Back to Carol Beer

Carol Beer was funny because she was obviously human. Her blankness was a performance, her indifference a choice, her “computer says no” a deflection she was consciously making. Behind the mask was a person who could, in principle, have done things differently.

The real version is less reassuring precisely because it lacks that quality. Nobody is being deliberately obstructive when HireVue tells a deaf applicant to improve their listening. Nobody chose to discriminate when the resume screener ranks a CV lower because the applicant’s name doesn’t fit the training data’s pattern. The system was built, trained, deployed, and is now running at scale — and when you ask why it decided what it decided, the honest answer is: we’re not entirely sure. The patterns were in the data. The data shaped the output. The output shaped your life.

Computer says no. And unlike Carol Beer, it cannot even be embarrassed about it.

The question, which is ultimately the question this book has been asking from its first page, is whether we notice this while there is still time to design things differently. The sketch was a warning dressed as a joke. Most warnings dressed as jokes are funnier than the thing they’re warning about.