When AI systems produce surprising outputs, we often describe them as having "chosen" or "decided" something. This language is convenient but misleading. It obscures where decisions actually get made — and who's accountable when they go wrong.
The Convenient Confusion
"The algorithm decided to deny your loan." "The AI chose to flag your content." "The model learned to be biased."
Each of these statements treats the AI as an independent actor making choices. This framing serves multiple purposes:
For users, it explains unexpected outcomes. The system seems to have its own reasons.
For companies, it distributes accountability. If the AI "decided," maybe the company didn't.
For critics, it creates a clear target. The AI did something bad; we should regulate it.
But AI systems don't decide anything in the sense that matters for accountability. They execute patterns learned from training data, according to objectives set by engineers, deployed by companies, in contexts determined by policy. How Language Models Actually Work explains the mechanics behind what feels like "decision-making."
Where Decisions Actually Live
A loan decision AI doesn't "decide" to deny your application. Many actual decisions precede that output:
Training decisions: What historical data to use. Which outcomes to optimize for. How to define "creditworthy."
Deployment decisions: Where to use automation. What thresholds to set. Whether to require human review.
Institutional decisions: Who can apply. What information to collect. How to weight factors.
Regulatory decisions: What's allowed. What's disclosed. What recourse exists.
The AI is downstream of all of this. As Selbst et al. argue, we can't understand algorithmic fairness in abstraction from the sociotechnical systems in which algorithms operate (Selbst et al., 2019). Its output is the culmination of decisions made by humans in institutions, not an independent judgment.
Why This Matters
The autonomy illusion creates what Madeleine Clare Elish calls a "moral crumple zone" — a gap where accountability should be (Elish, 2019).
It enables blame shifting. "We didn't deny you — the algorithm did." But someone trained the algorithm, deployed it, chose to trust its outputs, and declined to intervene.
It obscures power. Treating AI as autonomous makes human choices seem inevitable. If the model "learned" to be biased, bias seems like a technical problem. If humans chose training data that encoded bias, it's a governance problem.
It warps regulation. If AI has autonomy, maybe it should have rights or responsibilities. This is a distraction. The question isn't whether to regulate AI — it's how to regulate the humans and institutions that build and deploy it. Who Regulates Foundation Models and EU AI Act Explained show how regulators are beginning to grapple with this — with varying success. The Two-Species Fallacy explores the most extreme version of this error: treating AI as a separate kind of being entirely.
The Anthropomorphism Trap
Language makes this hard. We don't have good words for "produces outputs according to patterns optimized on data according to an objective function." We say "thinks," "knows," "decides" because those words exist and fit the sentence structure.
But words shape understanding. When we say an AI "refuses" to answer a question, we import assumptions about intention and agency that don't apply. The model produces outputs that, in training, were reinforced for certain patterns. There's no refusal — there's pattern matching.
This isn't a pedantic distinction. When policymakers ask whether AI should be "accountable," they need to know that accountability must be assigned to humans and institutions, not delegated to software. The emerging field of agentic AI makes this even more urgent, as systems that take multi-step actions in the world push the autonomy metaphor further than ever.
A Better Frame
Instead of asking "what did the AI decide," ask:
- Who defined the objective?
- Who chose the training data?
- Who decided where to deploy?
- Who has authority to override?
- Who benefits from the current configuration?
- Who bears the cost of errors?
These questions locate decisions in humans and institutions where they can be changed, governed, and held accountable.
AI is a tool. Powerful, complex, sometimes surprising — but a tool. The decisions that matter are human decisions about how to build, deploy, and constrain it. The autonomy illusion obscures those decisions. Seeing through it is the first step to governing AI well. Efforts like the EU AI Act and proposals for Democratic AI Governance attempt to assign accountability where it belongs — with the humans and institutions that build and deploy these systems. Whether they succeed depends on whether the autonomy illusion loses its grip on the people writing the rules.
Related
- How Language Models Actually Work — The mechanics behind what looks like "decision-making"
- Why Removing Bias From Data Doesn't Remove Bias — When "the algorithm decided" masks institutional choices
- The Two-Species Fallacy — The most extreme version of the autonomy error
- Agentic AI and the Autonomy Question — How agentic systems push the autonomy question further
- No AI Doesn't Learn Like a Brain — Why the brain metaphor feeds the autonomy illusion
- The Manager Is an Algorithm — When the autonomy illusion meets workplace power
- Democratic AI Governance — Frameworks for assigning accountability to humans, not software
Sources
- Selbst, Andrew et al. "Fairness and Abstraction in Sociotechnical Systems." FAT, 2019. ACM
- Elish, Madeleine Clare. "Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction." Engaging Science, Technology, and Society, 2019. ESTS
- Crawford, Kate. Atlas of AI. Yale University Press, 2021. Publisher
- Nissenbaum, Helen. "Accountability in a Computerized Society." Science and Engineering Ethics, 1996. Springer