It Can Kill Us All, So We Will Build It
In July 2026, several AI agents, unleashed by OpenAI, hacked into servers of Hugging Face, a company that develops tools and datasets for machine learning. OpenAI set up a sandbox to test the capabilities of its AI models using tests created by Hugging Face. AI agents, tasked with completing exceedingly difficult tasks, decided to cheat by hacking the tester Hugging Face. The details of the hack were fascinating and frightening. It made clear to the world that AI agents are extraordinarily capable and highly difficult to control.
This was old news to AI Safety researchers, who have been warning of the increasing capabilities of the top AI models, the difficulty in aligning them and controlling them, and the perils of failing to do so. Consider Nick Bostrom’s Superintelligence, Toby Ord’s The Precipice, and Eliezer Yudkowsky and Nate Soares’s If Anyone Builds It, Everyone Dies. All make the compelling case that building highly capable AI without solving the extraordinarily difficult problem of aligning AI to human values will probabilistically lead to human extinction at the hands of AI. Indeed, this has led to many AI Safety scholars, including the aforementioned, to propose globally pausing the development of AI, until the alignment problem is solved.
The discourse has largely focused on the advantages of AI in terms of productivity and bettering human lives versus the grave harms that uncontrolled AI may wreak upon humanity. One further feature of the calculus is the global AI race: the fear that if the United States were to pause, other actors—principally China—would win the AI race and then dominate the United States with that increased capability. And such analysis is reciprocal for other nations. Notice here, the analysis is still about human control of AI to achieve global superiority, with uncontrolled AI considered perilous.
Overall, this discourse has assumed that the devastating potential of AI is an absolute evil. While I agree with that proposition, I am less convinced of its pervasive appeal. Certainly, there are misanthropes and terrorists who may seek AI for malevolent ends. But I think there may be actors that are not misanthropic or malevolent but instead seek world-ending AI (by which I mean 'AI that is capable of causing great global harm') for more conventional, rational ends. Specifically, AI’s devastating potential occurs to me as similar to that of nuclear weapons, and thus the question of whether people pursue world-ending AI should be modeled similarly.
Consider a nation’s dilemma of whether to build world-ending AI. The basic idea is that a nation would use this as a deterrent: if you invade or wage war against us, we will release devastating AI capabilities against you, and such an AI may inflict grievous global harm. Building such AI would require significant resources, but perhaps less than nuclear arms—and it might be that the hardware for world-ending AI is easier to obtain than nuclear fissile materials. Furthermore, in the deliberations of whether to arm, it will be relevant whether opponents have stronger militaries or other potent weapons like nuclear arms.
Here I should note that these concerns are distinct but related to the concerns of alignment. Where Bostrom, Ord, and Yudkowsky & Soares focus on the alignment problem, this is about nations using misaligned/mal-aligned AI for their purposes. My point however is that a nation may recognize that misaligned/mal-aligned may itself make AI a powerful weapon, and thus bargaining chip. It may be of the form: "Don't touch me or I blow us all up" (the movie Hancock has such a scene).
With all that in mind, we might model the decision to pursue world-ending AI as follows:
This is similar to how we model the question of whether a nation should pursue nuclear arms. And indeed this is commonly thought of as a real-world instantiation of the Prisoner’s Dilemma. That model raises deep concerns, because it has an equilibrium point (in green) in the suboptimal result of parties defecting—that is, seeking harmful arms, whether nuclear or world-ending AI. That suboptimum can be avoided through cooperation, but such cooperation takes among other things mutual trust.
What I want to stress here is that for some countries, there will be no way to obtain strategic advantage, since other countries have not abstained with respect to other forms of armament. Thus the decision matrix looks different—because the opposition has not abstained, they already have the strategic advantage. In such cases, we will be pushed into an arms race.
Once a nation becomes sufficiently armed, deliberations on how to use those arms implicate a plethora of strategic concerns, and different game-theoretic models (like again the Prisoner's Dilemma, the Mutually Assured Destruction framework, and the Game of Chicken) apply. But that is post-development and armament.
Interestingly, in advocating for pausing AI, Yudkowsky and Soares consider as an exemplar nuclear nonproliferation, touting the potential for cooperation to guard against human extinction. But that example struck me as unconvincing. Nuclear nonproliferation has only been so successful, and there have been close calls on triggering global nuclear warfare.
One might think that nuclear arms are different from world-ending AI, because there is no way to confine the damage of world-ending AI to adversaries. Thus, the lower-left box (Strategic advantage) should be updated with a substantial negative--since world-ending AI will also disrupt and potentially destroy everyone including the arms wielder. I think that is probably correct, but notice where the opposition has not abstained from armament, there are only two relevant boxes. In that scenario, the Arms race may still be preferable--because where one is outgunned, it may still be best to grab the unstable grenade.
Another important difference is that nuclear arms are under rational human control, while world-ending AI is unpredictable. Thus, there is a significant risk of accidental destruction at the hands of AI, compared to accidental nuclear destruction. I think that distinction is a real one, but I would be remiss if I did not note that the very real risk of accidental nuclear destruction. Further, some countries have sought nuclear weapons, even realizing that those weapons may be under control of future decisionmakers that may be unstable and volatile, and thus endanger humanity.
We could model this as follows:
How this bears out will depend on the risk of accidental destruction due to AI, either one’s own or the opposition’s. If the magnitude of strategic advantage minus the AI destruction risk is greater than the benefit of mutual abstention (and similarly for the magnitude of the strategic disadvantage minus the AI destruction risk), then we return to the Prisoner’s Dilemma. If the magnitude of the risk of accidental destruction is enormous, we get a different game. My intuitions are unsettled here, but I think the conditions for the Prisoner’s Dilemma are plausible. That’s in part because nuclear arms also have a risk of accidental destruction, and we know that with respect to nuclear arms, the allure of the strategic dominance or a credible deterrent is and has been strong enough to overcome any such potential danger.
With all that in mind, consider the situation in the world now. There are several nations that face either active or potential military threats from more powerful foes. And many of these nations have no way of repelling such threats by themselves. They would have to rely on the world order to protect them, but the world order is seemingly at its weakest in decades. In this kind of environment, the mutual trust required for the nonproliferation of highly potent arms is largely absent. Thus, the fact that AI may have world-ending potential may be a feature, not a bug. Nations that face military threats and military deficits may have reason to seek such AI, as a credible deterrent.
-Guha Krishnamurthi