Something unusual happened in artificial intelligence recently. The people racing hardest to build the world's most powerful AI systems started publicly talking about slowing down — and talking to each other about safety.
Anthropic CEO Dario Amodei called for deliberately pacing frontier AI development. OpenAI CEO Sam Altman supported stronger independent evaluation, while major AI companies have begun discussing greater coordination around safety standards. Days earlier, Jacob Coxon, a researcher who worked on pretraining at both OpenAI and Anthropic, resigned from the latter and warned about the race toward increasingly capable and potentially self-improving systems.
I don't know whether Coxon is right about where this ends. Neither does he, Amodei, Altman or anyone else. But the people closest to frontier AI know considerably more than most of us about how it is beginning, and if some of them are now saying capability may be advancing faster than our ability to control it, we should take their warnings seriously.
The question isn’t whether AI is good or bad or whether America or China can build the best model. The more fundamental question is what happens if we actually succeed in building the intelligence we say we want to build.
What Are the Primary Safety and Control Challenges for Advanced AI?
- Declining Human Intervention: AI development prioritizes greater autonomy, self-correction and independent task execution.
- Reproducible Intelligence: Digital intelligence can be multiplied at scale without human biological limits.
- Institutional Lag: Machine intelligence operates at speeds faster than human legislative and regulatory bodies can supervise.
- The Coordination Paradox: Competitive market pressures, capital investment and national security incentivize companies to accelerate development despite safety risks.
What Did the Nuclear Age Teach Us?
The closest precedent for our current discussion is the nuclear age. In 1945, humanity acquired the technological capacity to destroy itself. What followed was an elaborate architecture of deterrence, engineering, diplomacy and fear designed to keep that capability under control.
Mutually assured destruction made a first strike potentially suicidal. Arms-control agreements created limits and verification. Hotlines reduced the possibility that misunderstanding led to escalation. Command-and-control systems created layers of authentication and authorization. The nuclear football became the most recognizable symbol of a larger architecture built around one crucial principle: An identifiable human decision must remain inside the chain.
That matters because deterrence ultimately depends on humans being deterrable. A leader contemplating a nuclear strike understands that another human may respond in kind. Fear, self-preservation and rational calculation become part of the control system.
The architecture is imperfect. Nuclear proliferation remains one of the world’s great unresolved dangers. But nuclear weapons have two properties that make control possible: They are physical, and they aren’t intelligent.
A nuclear weapon cannot decide it needs another weapon. It cannot copy itself onto thousands of servers, search for vulnerabilities in its command structure, persuade someone to give it additional resources or spend the night working on a better version of itself. Most importantly, it cannot independently decide to act. A human has to enter the chain.
AI obviously can’t do all those things today, and pretending otherwise turns a serious argument into science fiction. But look at what the industry is explicitly trying to achieve: greater autonomy, longer independent tasks, better coding, scientific research, tool use, self-correction and eventually AI systems capable of materially contributing to the development of better AI systems.
Each of those advances can create enormous value. Each can also reduce the amount of human intervention required between intelligence and action. Nuclear weapons gave humans the ability to destroy civilization. Advanced AI raises a different control problem: the possibility of increasingly consequential capability that requires less and less human authorization.
The nuclear age taught us how to control an extraordinarily destructive tool. The AI age may require us to learn how to control something that no longer behaves merely like a tool.
What Happens When Einstein Can Be Copied?
There is another difference between AI and almost every previous technological revolution: Intelligence has always been scarce.
Albert Einstein had one brain. He needed sleep. He could work on only so many problems and collaborate with only so many people. Most importantly, Einstein couldn’t copy himself.
Digital intelligence has very different economics. Imagine — not today, but on the trajectory the industry itself is pursuing — that we create an AI researcher with the intellectual capacity of an Einstein, John von Neumann or Richard Feynman. Creating one would be extraordinary. The real transformation begins when we can create a thousand.
Put 200 on physics, 200 on biology, 200 on materials science, 200 on computer science and another 200 on improving AI itself. They could work continuously, exchange discoveries almost instantly and potentially incorporate improvements across every copy. When one becomes materially better, that version can be replicated again.
This is why “superintelligence” may actually be too narrow a description. We tend to imagine one giant digital brain becoming smarter than humanity. The more consequential transformation may be the industrialization of intelligence itself: exceptional cognitive capability becoming reproducible at declining marginal cost and operating in parallel at a scale biology could never provide.
The upside is extraordinary. So is the governance problem. Human civilization has always been constrained by the scarcity of exceptional minds; if exceptional intelligence becomes software, the limiting factors may increasingly become compute, chips and electricity. Unlike Einstein’s brain, we know how to manufacture more of those.
Can Human Institutions Keep Up With AI?
Human institutions operate at human speed. Congress can take years to legislate, and international treaties take longer still. Regulators consult, courts deliberate, boards meet and scientific communities test and challenge one another's work. Those processes can be frustratingly slow, but they’re how humans make consequential decisions.
Machine intelligence operates on another clock. If thousands of capable AI agents can eventually conduct research, write code, test hypotheses and communicate at machine speed, human oversight faces a mathematical problem before it faces a political one. Humans don’t suddenly become less intelligent; biology simply has a speed limit.
That is what I mean by a potential point of no return. It doesn't require an AI becoming conscious or doing something cinematic. It may simply be the point at which machine-generated capability advances faster than human institutions can understand, evaluate and meaningfully supervise it.
Why Doesn’t Everyone Just Slow Down?
This is what makes the events of the past weeks so important. Amodei wants frontier development paced. Altman supports stronger independent evaluation. Competing AI laboratories are discussing common safety standards. Some of the fiercest competitors in technology are beginning to acknowledge that certain risks may require cooperation.
That is encouraging, but it also exposes the central paradox. If Anthropic substantially slows while OpenAI continues, Anthropic risks losing. If they both slow while Google, xAI or another competitor continues, they risk losing together. If every American laboratory slows while China continues, the problem becomes geopolitical.
Safety says slow down. Capital says accelerate. National security says don’t let the other country get there first. Every participant can therefore behave rationally while collectively producing an outcome none of them actually wants.
This isn’t necessarily hypocrisy. It is a coordination problem, and desire for the prizes at stake makes coordination exceptionally difficult. Advanced AI could confer scientific leadership, military advantage, enormous economic power and control over perhaps the most productive resource civilization has ever created: intelligence itself.
Open-weight AI adds one more complication. China matters here not because Chinese AI is uniquely dangerous, but because no country can solve this problem unilaterally. Once sufficiently capable model weights are widely distributed, governments and frontier laboratories may no longer be the only actors capable of advancing or deploying them.
Nuclear proliferation required uranium, enrichment facilities, specialized engineering and physical supply chains. AI proliferation increasingly requires compute and software. Open weights can turn an AI race into a proliferation problem, and proliferation makes control much harder.
Is This Really an AI Problem or a Human One?
For thousands of years, humans have imagined beings possessing intelligence and power vastly greater than our own. We called them gods. The Tower of Babel is one of our oldest stories about the same instinct: Humanity builds upward, attempting to reach heaven, until its ambition exceeds its limits.
I don’t take technology policy from Genesis, but I understand why this story survived. It isn’t really about a tower. It’s about the human difficulty accepting that something should remain beyond our reach.
For most of history, our ambition exceeded our technology. AI raises the possibility that our technology may finally catch up with our ambition and perhaps surpass the intelligence that created it.
That is the philosophical discontinuity underlying this debate. The steam engine became stronger than us. Airplanes became faster. Computers calculated better. But humans remained the highest general intelligence in the system. AI is the first technology whose stated development trajectory potentially challenges that hierarchy.
If we eventually build something substantially more intelligent than ourselves, what exactly is our relationship to it? Tool, partner, creation, successor? We don’t have a good word because we’ve never had the problem.
What Would Make AI Optimism Rational?
I want to be an AI optimist because the upside is almost impossible to overstate. Imagine thousands of extraordinary researchers working on cancer, Alzheimer’s, clean energy, materials and food production. Imagine every child having an exceptional tutor and every physician having access to the accumulated expertise of medicine.
If we’re going to create thousands of Einsteins, there are thousands of problems I desperately want them solving.
And something encouraging did happen this week. Competitors began talking a little more like custodians of a shared risk and a little less like contestants trying to win a prize. Arms-control regimes also began when adversaries recognized that some outcomes were disastrous regardless of who technically won.
But cooperation isn’t control. Independent evaluators, common standards and international agreements can buy us time, and Amodei is right that we should buy it. Time matters only if we know what we’re buying it for.
We need measurable capability thresholds, independent testing, real fail-safes, limits on autonomous access to critical physical and digital systems and eventually international agreements around capabilities that no country should want proliferating. Most importantly, we need a credible answer to a deceptively simple question: Where is the human chokepoint?
Nuclear strategy spent eight decades constructing systems intended to ensure that, however destructive the weapon became, identifiable human decisions remained inside the chain. AI development is moving in almost the opposite direction. We celebrate greater autonomy, longer independent tasks, self-correction, AI conducting research and eventually perhaps AI helping create better AI.
Each advance may produce enormous benefits. Each is commercially rational. But each can also remove another human decision from the loop.
Maybe reproducible intelligence becomes humanity’s greatest achievement. Maybe it unlocks a century of scientific progress and abundance that we can barely imagine. I genuinely hope that’s where we're going.
But hope isn’t command-and-control. Before thousands of Einsteins can work at machine speed — and before some of those Einsteins are assigned to building their successors — we need to answer one question:
If one day we need to tell it no, what makes us certain it will still listen?
