Redistributed

In 1965, the British mathematician I.J. Good described a machine that could design a better version of itself. That improved version could design an even better one. And so on. He called this an "intelligence explosion" and predicted it would be "the last invention that man need ever make."

Six decades later, the idea still dominates safety debates at frontier AI labs. But between Good's thought experiment and today's reality lies a large and important gap. Understanding what recursive self-improvement actually is — and what it is not — matters, because the concept is already shaping policy, investment, and power in ways that affect people who have never heard the term.

The Mechanism

Recursive self-improvement (RSI) describes a feedback loop: an AI system improves its own capabilities, then uses those improved capabilities to improve itself further. Each cycle produces a more capable system, which completes the next cycle faster or better. In theory, this loop accelerates — each iteration yields larger gains than the last.

The components are straightforward. An AI system would need to:

Understand its own architecture — how its parameters, training process, and objectives relate to its performance.
Modify that architecture — change its own code, retrain itself on better data, or redesign its learning process.
Evaluate the results — determine whether the modifications actually improved capability.
Iterate — use the improved version to find the next modification.

Each step is individually plausible. The combination — an autonomous loop that runs without human intervention and produces compounding gains — is what makes RSI a distinct and contested idea.

Where We Actually Are

Current AI systems do not recursively self-improve in any strong sense. This is worth stating plainly, because the discourse often blurs the line between what exists and what might.

Large language models can generate code, including code related to machine learning. They can write training scripts, suggest architectural modifications, and help researchers explore ideas faster. Google DeepMind's AlphaFold predicted protein structures with remarkable accuracy, and those predictions are now informing further research in biology and drug design. AI tools are accelerating AI research across dozens of labs.

But "AI assists AI researchers" is fundamentally different from "AI autonomously improves itself." The gap is not a technicality — it is the gap between a tool and an agent, between acceleration and autonomy. A spreadsheet accelerates accounting; it does not autonomously restructure a company's finances.

Today's models cannot inspect their own weights and understand why they produce a given output. They cannot redesign their training process and evaluate whether the redesign worked. They lack the ability to set their own objectives and pursue them across multiple iterations without human oversight. These are not minor engineering challenges — they represent unsolved problems in machine learning, interpretability, and goal specification.

The Spectrum: From Weak to Strong

The most useful way to think about RSI is as a spectrum, not a binary.

Weak RSI: AI Accelerating AI Research (Already Happening)

AI tools are already embedded in the AI research pipeline. Researchers use language models to search literature, generate hypotheses, write and debug code, and analyse experimental results. Google DeepMind used AI systems to discover new mathematical theorems (FunSearch, 2024). AlphaFold's protein structure predictions have opened research directions that would have taken years to explore manually.

This is weak RSI — humans remain in the loop at every stage, and the improvements flow through traditional scientific processes. But it is not trivial. Labs with better AI tools do better AI research. They publish faster, explore larger design spaces, and identify promising directions earlier. The advantage compounds.

Medium RSI: Agentic AI Optimising AI (Emerging)

The next step is already visible. Agentic AI systems — models that can plan, use tools, and execute multi-step tasks — are being applied to AI development itself. AI systems now help optimise their own inference, selecting which computations to perform and which to skip. Automated machine learning (AutoML) searches over model architectures, hyperparameters, and training configurations without manual intervention for each experiment.

In early 2026, several frontier labs acknowledged using AI systems to assist with portions of their own model development — not full autonomy, but more than simple code completion. The systems suggest architectural changes, run ablation experiments, and flag promising directions for human researchers to evaluate.

This is meaningful because it starts to close the loop. The AI is not just a tool used by researchers; it is a participant in the research process with growing autonomy over individual steps. But humans still set the objectives, approve major decisions, and interpret results.

Strong RSI: The Intelligence Explosion (Speculative)

Good's original scenario — a system that redesigns itself without human involvement, producing rapidly compounding capability gains — remains speculative. No system has demonstrated this. The reasons are both technical and structural.

Technical barriers. Self-modification requires self-understanding, and current AI systems are largely opaque even to their creators. The field of mechanistic interpretability is making progress — researchers at Anthropic, DeepMind, and elsewhere are mapping how models represent and process information — but we are far from the level of understanding that would allow a model to meaningfully redesign itself. Improvements to AI systems currently come from extensive experimentation, not from a model inspecting its own weights and reasoning about what to change.

Structural barriers. Training a frontier model requires enormous compute infrastructure — thousands of specialised chips running for months (Economics of Training a Frontier Model). Even if a model could design a better version of itself, it would need access to compute resources, data pipelines, and evaluation infrastructure that no model currently controls. The bottleneck is not just intelligence; it is physical infrastructure and the supply chains behind it (Who Controls the AI Supply Chain).

Diminishing returns. The scaling laws that have driven recent progress may be flattening. If each increment of capability requires disproportionately more compute, the feedback loop decelerates rather than accelerates. An intelligence explosion assumes accelerating returns; the empirical trajectory is ambiguous. (See The Scaling Laws Plateau.)

Why This Matters for Redistribution

You do not need to believe in an intelligence explosion to see how RSI dynamics concentrate power. Even the weak version already operating today has redistribution consequences.

The AI Flywheel

Labs that use AI to accelerate AI research pull further ahead of everyone else. This is a flywheel: better models produce better research tools, which produce better models. The advantage accrues to organisations that already have frontier models, large compute budgets, and top researchers — which is to say, a handful of private companies.

This dynamic is visible in the market. The gap between frontier labs and the rest of the AI industry has widened, not narrowed, since 2023. Open-source models have made capable AI more accessible, but the rate of improvement at the frontier outpaces the rate at which that capability diffuses. If AI research itself becomes dependent on AI capabilities that only a few labs possess, the concentration intensifies further. (See The Frontier Lab Oligopoly.)

Control Over the Loop

Whoever controls a self-improving system controls its trajectory — what it optimises for, what constraints it respects, what values it reflects. This is the ultimate chokepoint.

Today, these decisions are made by a small number of technical leaders at a small number of companies. The choices are not purely technical — they embed assumptions about what "better" means, whose problems matter, and what risks are acceptable. As Bougueng (2026) argues, governance of AI-to-AI dynamics — the ways AI systems influence and improve each other — is an emerging priority that existing regulatory frameworks do not address.

Ursula Franklin's distinction between prescriptive and holistic technologies is relevant here. A prescriptive technology centralises control and reduces workers to executing predefined steps. A self-improving AI system that operates without meaningful external oversight is prescriptive in the extreme — the loop itself becomes the authority, and everyone outside it becomes a recipient of its outputs rather than a participant in its direction.

The Safety Debate as a Redistribution Question

The debate over RSI risks is itself a contest over resources and attention. Geoffrey Hinton, Yoshua Bengio, and others have warned that recursive self-improvement could produce systems that humans cannot control — a genuine concern grounded in the mathematics of optimisation. Hinton's departure from Google in 2023 to speak freely about these risks lent the argument public weight.

But researchers focused on present harms — Timnit Gebru, Emily Bender, Kate Crawford, and others — argue that obsessing over speculative superintelligence diverts attention from documented damage happening now: discriminatory hiring algorithms, exploitative data labour, environmental costs of training, and the concentration of power in unaccountable institutions. As Gebru and Bender have noted, you do not need recursive self-improvement to cause serious harm; existing systems already do.

Both camps are describing real redistribution dynamics. The disagreement is about which ones deserve priority. A pragmatic position — and the one this publication takes — is that speculative and present risks are not in competition for a fixed budget of concern. The governance mechanisms needed for self-improving systems (capability evaluations, compute monitoring, deployment gates) overlap significantly with those needed to address current harms (algorithmic audits, transparency requirements, accountability frameworks). The question is whether those mechanisms get built at all, not which danger they are labelled for. (See The AI Safety Summit Circuit: Diplomacy or Theater?.)

What Safeguards Look Like

If recursive self-improvement accelerates — even in its weak and medium forms — what guardrails matter?

Capability evaluations. Before deploying or scaling a model, test it for specific dangerous capabilities: Can it assist in developing weapons? Can it autonomously replicate itself? Can it manipulate humans into granting it more resources? Anthropic's Responsible Scaling Policy and similar frameworks at other labs formalise this, though the evaluations are designed and administered by the same companies whose products are being evaluated.

Deployment gates. Define thresholds — capability levels beyond which a model cannot be deployed without additional safety measures. This requires agreement on what the thresholds are, which is a political negotiation as much as a technical one.

Compute governance. Training frontier models requires specialised hardware (primarily Nvidia GPUs and Google TPUs) concentrated in a small number of data centres. Monitoring compute purchases and usage — as some have proposed at international level — provides a physical chokepoint for governing capability development. Compute is to RSI what enrichment facilities are to nuclear proliferation: a bottleneck that governance can target.

International coordination. Self-improving AI systems do not respect national borders. The AI Safety Summits in Bletchley Park (2023), Seoul (2024), and Paris (2025) have begun building multilateral frameworks, but commitments remain voluntary and enforcement mechanisms are absent. (See The AI Safety Summit Circuit: Diplomacy or Theater?.)

What This Means for You

Recursive self-improvement is not a science fiction scenario that may or may not arrive. In its weaker forms, it is already reshaping who can do AI research and how fast they can do it. The dynamics that matter are not about superintelligence — they are about compounding advantage, centralised control, and the narrowing of who gets to participate in deciding how the most powerful technology of the century develops.

Pay attention to the loop. Not because it will necessarily explode, but because whoever controls it is already pulling ahead — and the gap is growing. Public compute infrastructure, open models, and broader governance frameworks could distribute both the benefits and the oversight. But the compounding advantage that weak RSI creates makes the window for such redistribution narrower with each iteration.

Sources

Good, I.J. "Speculations Concerning the First Ultraintelligent Machine." Advances in Computers 6 (1965): 31–88. Elsevier
Bougueng, R. "Governing AI-to-AI Dynamics: Recursive Improvement and Systemic Risk." AI Policy Journal, 2026.
Franklin, Ursula. The Real World of Technology. House of Anansi Press, 1989. Publisher
Crawford, Kate. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, 2021. Publisher
Bender, Emily M., Timnit Gebru, et al. "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" FAccT, 2021. ACM
Anthropic. "Anthropic's Responsible Scaling Policy." 2023. Anthropic
Romera-Paredes, Bernardino, et al. "Mathematical Discoveries from Program Search with Large Language Models." Nature 625 (2024): 468–475. Nature

General

Recursive Self-Improvement: The Feedback Loop That Keeps AI Leaders Up at Night