In their landmark book The Golem: What You Should Know About Science (1993), Harry Collins and Trevor Pinch popularised the concept of the experimenter’s regress to describe a fundamental problem in scientific validation. When scientists develop a new experiment or instrument, they face a circular challenge: they can only know their experiment works properly if it produces the right result, but they can only know they have the right result if the experiment is working properly. This creates a logical loop with no clear exit point. The experimenter must eventually make a judgment call, often by comparing their results to others’ work or by relying on theoretical expectations, breaking out of the regress through social consensus rather than scientific logic.
An analogous regress, it turns out, can be observed in the construction of contemporary AI decision systems, in ways that reveal limitations in how we deploy algorithmic governance. In fact, two different versions of this circularity trap open up in front of those who would deploy AI decision tools. As organisations increasingly delegate high-stakes decisions to AI systems or involve them in their decision processes, from loan approvals to hiring recommendations to criminal sentencing, we find ourselves trapped in variations of the experimenter’s regress that should give us pause.
The First Regress: Replicating the Past
Consider the most common approach to building AI decision systems: training them on historical human decisions. This is quite natural to computer scientists trained in the machine learning mode: a supervised learning system is trained on examples in which a human chose different decision options, developing a model allowing it to correctly predict the human decision within the human decision sample, and then extrapolates from there to unseen decisions. We might compile vast datasets of past choices, such as which job applicants were hired, which loan applications were approved, which parole requests were granted, and use them to teach algorithms to predict the human decision-makers’ behaviours and thus automate them. The system learns to replicate human decision-making at scale and speed.
But here’s where the first, simplest regress emerges. How do we know if our AI system is working correctly? We test it against human decisions. If it accurately predicts what a human decision-maker would have done, we declare success. The algorithm has learned to make decisions “correctly.”
Yet this validation process assumes that the historical human decisions were themselves correct: an assumption we know to be false. Human decision-making is riddled with biases, inconsistencies, and errors. Loan officers have discriminated based on race and gender. Hiring managers have favoured candidates reminiscent of themselves. Judges have given harsher sentences before lunch when they are hungry. If these flawed decisions constitute our training data and our validation benchmark, we are essentially calibrating our AI system to reproduce human fallibility.
We have not exited a circularity loop, only created a simplified experimenter’s regress: we determine that the AI is making good decisions because it matches human decisions, and we trust human decisions as our benchmark because that is the data set we have. Like Collins and Pinch’s experimenter who validates their apparatus by getting the same results as another experimenter (who may have a similarly miscalibrated instrument), we are potentially creating systems that reliably reproduce systematic errors at scale.
The danger here is not just replication, it is amplification. When we encode biased patterns into algorithmic systems and then deploy them across millions of decisions, we risk transforming idiosyncratic prejudices into systematic error. The AI does not just learn what decisions humans made; it learns to make those decisions with mechanical consistency, stripping away even the possibility of human second-guessing or case-by-case compassion.
The Second Regress: The Better Decision Problem
Perhaps recognising this limitation, we might aspire to something more ambitious: AI systems that make not just faster decisions, but better ones. These decisions would transcend human biases and limitations, or at least outperform the baseline levels of human fallibility. This is where we encounter an even more intractable analogue of the experimenter’s regress, more closely attuned to the one Collins and Pinch identified in scientific research.
If we want AI to make better decisions than humans, we face a fundamental validation problem: how do we know the decisions are actually better? We cannot simply compare the AI’s outputs to human choices, because we are explicitly trying to improve upon those choices. We need some independent standard of “correct” or “good” decisions.
But for most complex decisions—hiring, lending, sentencing, medical treatment—no such objective standard exists. What is the “right” decision when choosing between job candidates with different strengths? What is the “correct” sentence for a particular crime? These are normative questions saturated with values, trade-offs, and contestable priorities. Where we cannot identify the fact of the matter, even through meticulous scrutiny or long-term follow-up, we cannot establish that the AI outperforms the human.
We have one type of use case where the AI might add some value here. If we believe that a human who is paying careful and detailed attention will outperform a human who is deciding at speed, then we could train the AI to replicate the careful and diligent human decision-maker and implement the AI tool at speed. This gets us closer to a best-of-both-worlds scenario, in which the more diligent decision-making process can be replicated at speeds that outpace even the slapdash human.
But this case functionally lapses back into the first regress: we are only really able to trust this AI’s superior decision-making insofar as we trust the diligent human standard, and we still find ourselves without a yardstick of decision quality by which to defend the claim that the diligent decision-maker actually decides better. We might be in the dangerous alternative case in which diligent thought overrides intuitions to the detriment of decision quality.
Unable to directly validate that AI decisions are better, we are forced to fall back on trust in the process: we must believe that the developers, through their choice of training data, optimisation objectives, and algorithmic architecture, have successfully created a system that makes superior decisions. But this is parallel to the experimenter’s regress. We are trusting the “experiment” (the AI system) without being able to independently verify its “results” (better decisions).
This situation creates space for what Meredith Broussard, in More Than a Glitch (2023) calls “technochauvinism”. This is the belief that technology is always superior, that algorithmic decision-making must be better simply because it is AI-driven, computerised: i.e., objective. Technochauvinism fills the validation gap left by the experimenter’s regress with an article of faith: the AI is making better decisions because AI systems are inherently more rational, objective, and sophisticated than human judgment.
But strip away this faith, and we are left with uncertainty. Is the AI making better decisions, or just different ones? Is it correcting human biases, or introducing new ones? Is it optimising for the right objectives, or efficiently pursuing goals that do not align with justice, fairness, or success? Without a way to definitively answer these questions, we cannot escape the regress.
Breaking the Loop
Collins and Pinch showed that scientists ultimately break out of the experimenter’s regress through social processes: debate, consensus-building, and provisional agreements about what counts as a valid result. As John Turney (2020) puts it: “those who get it right succeed at least partly because they get everyone else to agree with their definition of what getting it right means.” Read in some heavy scare-quotes for that first instance of ‘get it right’. The same may be necessary for AI decision systems where humans cannot independently determine the objectively correct decision.
Rather than treating algorithmic decision-making as a purely technical problem with technical solutions, we have to recognise it as fundamentally social and political. This means ongoing public scrutiny, democratic oversight, and frank acknowledgment that AI systems embody contestable values and trade-offs rather than objective truth.
It means accepting that we cannot outsource the responsibility for difficult decisions to algorithms, because validating those algorithms requires the very judgment we were hoping to avoid. The experimenter’s regress reminds us that there is no escape from human judgment in socially-embedded high-stakes decision making, only choices about where and how to exercise it.
The golem, in Jewish folklore, was a powerful servant that could protect or destroy depending on how it was commanded and controlled. Collins and Pinch used the metaphor to describe science itself: powerful, useful, but requiring careful guidance and inherently fallible. Our AI decision systems are golems too. Recognising the regress at their heart is an important element of wielding them well.
Works cited:
Broussard, M. (2023) More than a Glitch: Confronting Race, Gender, and Ability Bias in Tech (MIT Press)
Collins, H. & Pinch, T. (1993) The Golem: What everyone should know about science (Cambridge University Press)
Turney, J. (2020) ‘The Golem – SSK as pop science?’, Science Observed, available at: https://scienceobserved.wordpress.com/2020/12/02/the-golem-ssk-as-pop-science/