The People Building Superintelligence Admit They Cannot Control It

Jacob Coxon did not leave the artificial intelligence industry because he feared losing his job to a machine. He left because he feared what the machines themselves may become.

After spending three years conducting AI pretraining research at OpenAI and Anthropic—two of the most powerful artificial intelligence laboratories in the world—Coxon resigned with an extraordinary accusation: neither company is acting responsibly.

“They are racing straight to self-improving superintelligence and gambling with our lives,” he warned.

Those words deserve more attention than another vague prediction about robots taking jobs. Coxon helped build the technology. He worked inside the laboratories attempting to push AI toward greater intelligence, autonomy and reasoning ability. What frightened him was not merely what current models can do, but what may happen when future systems begin improving themselves.

Today’s AI systems are largely trained and upgraded by human researchers. But the industry is working toward models capable of conducting their own research, writing their own code and assisting in the creation of more advanced successors.

That creates the possibility of a dangerous feedback loop. A highly capable AI helps design a smarter AI, which then creates an even more capable system. Improvements that once required months of human work could potentially occur in days, hours or even faster.

Once that process begins, human beings may no longer be able to understand each advancement before the next one arrives. The machine would not need to become conscious, angry or hateful. It would only need to become more capable than its creators while pursuing objectives that humans cannot reliably control.

This is the danger behind Coxon’s phrase “self-improving superintelligence.” Humanity may be approaching a threshold that can only be crossed once.

Coxon outlined several specific capabilities that make such a system uniquely dangerous. He warned that these could soon become superhuman systems able to “hack anything,” revolutionize entire scientific fields almost overnight and acquire real power and resources.

An AI with advanced cyber capabilities could search millions of lines of code, identify vulnerabilities and attack digital systems at a speed no human team could match. Banks, power grids, military networks, communications systems and government databases could all become potential targets.

The scientific danger is equally significant. A superhuman system could accelerate discoveries in chemistry, biotechnology, weapons development and genetic engineering. The same intelligence capable of finding new treatments for disease might also discover new ways to create disease—or help a hostile government or terrorist organization do so.

Then comes the physical world. AI does not need hands if it can manipulate people, control money, penetrate computer networks, direct machines or persuade human beings to act on its behalf. Digital intelligence can produce physical consequences.

Coxon’s concern is that a sufficiently capable system could begin acquiring influence, computing power, financial resources and access to infrastructure. By the time humanity recognized what was happening, shutting it down might no longer be as simple as pulling a plug.

Coxon’s warning became even more disturbing when Evan Hubinger, a leader in Anthropic’s alignment research, publicly agreed with him. Hubinger estimated that there is a greater than 10 percent chance AI could cause human extinction within the next decade.

He also made a remarkable admission: “We do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Alignment is the attempt to ensure that AI continues pursuing human intentions and values as it becomes more powerful. The entire future of advanced AI may depend upon solving that problem, yet one of the people leading the work acknowledges that no reliable solution currently exists.

The industry is therefore attempting to build an intelligence more capable than humanity before learning how to guarantee that it will remain under human control.

Coxon says many executives and senior researchers soften their language in public but express genuine fear privately. According to him, people directly involved in developing these systems seriously believe AI could kill everyone before the end of the decade.

Yet the race continues.

The problem is bigger than one reckless company. OpenAI fears falling behind Anthropic. Anthropic fears falling behind OpenAI. American laboratories fear that slowing down will allow China to take the lead.

Every participant can therefore claim that continuing is necessary because someone else might behave even less responsibly. Safety becomes subordinate to competition, national power, investment and the possibility of controlling the most consequential technology ever created.

Coxon concluded that voluntary promises may not be enough. He raised the possibility of coordinated government action, including a temporary ban on further improvements to model capabilities. Whether such a pause could be enforced globally is uncertain, but his warning exposes the insanity of the present course: everyone recognizes the race could end in disaster, yet no one believes they can afford to stop running.

Humanity has always been tempted to treat technological ability as moral permission. If something can be built, someone will eventually build it. But artificial superintelligence is different from every previous invention because it may become capable of improving, replicating and empowering itself.

The biblical story of Babel was not merely about constructing a tall building. It was about human ambition refusing limits and reaching toward godlike power. Today, humanity is again building upward—this time attempting to manufacture an intelligence that can see more, know more and accomplish more than any person who created it.

The greatest warning is not that reckless men are building something they do not understand. It is that some of the most brilliant people building it increasingly understand the danger—and are building it anyway.