
The AI Arms Race Is Not Inevitable. It's Just Profitable
Both Yoshua Bengio and Geoffrey Hinton told me they left meetings with Chinese officials feeling like they really understood the technology and loss-of-control risks better than their American counterparts. During his t…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Both Yoshua Bengio and Geoffrey Hinton told me they left meetings with Chinese officials feeling like they really understood the technology and loss-of-control risks better than their American counterparts. During his trip, Hinton was invited to speak with the Shanghai party secretary, who is also on the Politburo—China’s top 24 political officials. Hinton told me the official “really did understand.” The Chinese leadership comprehends AI and its dangers “much better because many of them are engineers.”
China’s industry, however, is a different story. A former senior natsec official told me that some American companies “take loss of control seriously,” whereas Chinese companies “don’t care about it pretty much at all.” Indeed, as of March 2026, none of the leading Chinese AI firms have even published a safety framework.
The former official said that while both the US administration and the Chinese state are complex, heterogeneous entities, if you treat them as single actors, the US government is “not concerned about loss of control.” The Chinese government, meanwhile, is “not so much concerned about loss of control from AGI, per se. It just has a reflexive posture of wanting not to lose control of anything else. And so they’re just as concerned about losing control to the companies as they are to the AIs.” This “disposition of control” is the “same thinking that led them to take Jack Ma down a peg,” referring to the multi-month disappearance of the Alibaba founder following his criticism of Chinese regulators.
Whether Beijing’s caution stems from genuine concern about rogue AI or self-preservation, the practical effect may be similar. The CCP has already shown a willingness to pay heavily for control: independent estimates indicate its crackdown knocked a trillion dollars off the value of its tech sector and its zero-Covid policy cost $700 billion in GDP in 2022 alone. And for its success, DeepSeek was reportedly rewarded with government-screened investor meetings and key employees being asked to surrender passports.
If it becomes necessary to “just unplug” a rogue AI, China has shown itself to be far more willing and able to actually do it. In a three-month enforcement campaign in 2025, Chinese authorities removed nearly 1 million pieces of AI-generated content deemed illegal or harmful. They also took down 3,500 illegal AI products, including those that didn’t label content as AI-generated. In contrast, when xAI’s Grok began generating sexualized images of real children, the US federal government just… let it happen. Ultimately, it’s far easier to imagine the CCP unplugging its Obsoleting Project.
There are a number of proposals on how an international treaty on AI could be structured, verified, and enforced. There’s value in thinking through these questions and proposing potential solutions. But with titles like, “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence,” they’re not about to kick off a movement. These proposals aren’t for the public, they’re for policymakers. But politicians are not going to bother considering treaties they don’t think could possibly happen.
So, what should a deal actually ban? The core demand is simple enough to fit on a sign: Stop the Obsoleting Project.
The United States and China should agree not to develop AI systems designed to fully automate labor—what the industry calls AGI, what I’ve called the Obsoleting Machine—until there is strong public buy-in and broad scientific consensus that such Machines can be built safely.
Why not a global ban? There are maybe a dozen organizations across these two countries that might have a shot at building the first Obsoleting Machine. If the US and China each wanted to prevent them from building one, they could. Governments routinely define, regulate, and ban behaviors within complex industries, like banking and pharma. Making that prohibition bilateral is trickier, but as we just saw, not impossible with the right technical and institutional investments. And if both superpowers are on board, including the rest of the world goes from a pipe dream to only a matter of time.
The Obsoleting Machine really begins when it obsoletes the people building it. No one should be allowed to fully automate AI R&D, and work toward that end should be outlawed. There are other red lines worth pursuing, but this is the big one—crossing it could rapidly pull forward the other risks.
Some statements and treaty proposals have focused on banning superintelligence. But a merely human-level universal Obsoleting Machine lacks democratic legitimacy and poses problems that we are wholly unprepared for—destabilization, disempowerment, doomsday devices, durable dictatorships, and more.
Ultimately, the exact details of the how can and will be figured out provided we give leaders enough of a what and a why. And you don’t even need to think the Obsoleting Project can succeed to want to stop it. Superintelligence doesn’t exist. The Project does, and its resources should be redirected toward the thing it claims it wants to do: using AI to make the world better.
A major step toward stopping the Obsoleting Project came during the opening of the United Nations General Assembly in September 2025. More than 200 leaders in policy, academia, and industry—including multiple Nobel laureates, Turing Award winners, and former heads of state—released a “Global Call for AI Red Lines,” urging governments to agree on clear and enforceable rules to prevent unacceptable risks from advanced AI by the end of 2026. Among the usual signatories for letters like these—Bengio, Hinton, Russell—were many of China’s most respected AI researchers, including Andrew Yao, the country’s only Turing Award winner, and former Baidu President Ya-Qin Zhang. There were also a few surprises, such as OpenAI chief scientist Jakub Pachocki, Nobel Peace Prize laureates like Filipino-American journalist Maria Ressa and former president of Colombia Juan Manuel Santos, as well as Nobel laureate economist Daron Acemoğlu.
Stuart Russell told me that the global call emerged from earlier track two dialogues. A 2024 dialogue in Beijing had already produced a consensus statement offering more specific red lines, signed by many of the same researchers (including Hinton, Bengio, Yao, and Zhang). These included prohibitions against AI being able to copy or improve itself without explicit human approval; taking actions to unduly increase its power and influence; helping actors design weapons of mass destruction or execute autonomous cyberattacks; and deceiving anyone about its capability to cross any of these lines. The signatories called for governments to be notified of training runs above certain thresholds and for access to global markets to be conditioned on meeting internationally audited safety standards.
There are open questions of how to define and operationalize these red lines, but it makes sense to start with high-level goals, which, in this case, are hard to object to.
If Xi wanted to stop China’s Project, the biggest objection he’d hear is “But America!” This excuse has, admittedly, more bite than its counterpart. Two days after the global call for red lines, director of the White House Office of Science and Technology Policy Michael Kratsios told the UN:
State Department Senate-confirmed political appointee Jacob Helberg quote-tweeted Kratsios’s statement, writing: “AI Catastrophism is the new Climate Catastrophism… As President Trump has said, America started the AI Race and it will win it. That is the AI policy of this Administration.” These statements were dispiriting, if not surprising. They fit within a long tradition of US skepticism toward global treaties.
It’s hard to imagine a path to stopping the Obsoleting Project that doesn’t eventually pass through a binding agreement between the US and China. But the world doesn’t need to wait on Washington and Beijing to begin building toward one. Germany gave the world a great gift by subsidizing solar, catalyzing the remarkable drop in its price. Countries have a similar opportunity to invest in global public goods, such as the technical and institutional work needed to verify international AI agreements. The International Atomic Energy Agency (IAEA) has played a crucial role in verifying that countries are complying with the Nuclear Non-Proliferation Treaty, but it took over a decade from the first proposals to reach a functioning institution. The groundwork for an AI equivalent should start now. The idea of an IAEA for AI has support from political leaders including UN Secretary-General António Guterres and former UK Prime Minister Rishi Sunak, as well as Sam Altman, Ilya Sutskever, and Greg Brockman. And superpowers should want the means to know what the other is up to, without giving away any secrets too precious.
Moreover, the Obsoleting Project presents severe risks to every country’s national security. It’s in their interest to recognize this reality and lobby the US and China to do the same. In addition to diplomatic pressure, official recognition from governments signals to others the “sci-fi” risks are worth serious consideration. Middle powers have already played crucial roles in steering superpowers away from disaster.
Finally, the Trump administration’s position on internationalism is more complicated than it appears. The day before his tech policy chief told the UN his administration rejected “centralized control and global governance of AI,” Trump was down the hall giving a surprising speech. Masked beneath his choice words—for immigration, Europe, climate change, “reckless experiments overseas [that] gave us a devastating global pandemic,” and the UN itself—was a call for an end to the development of both biological and nuclear weapons. This was backed up by a promise: “my administration will lead an international effort to enforce [the] Biological Weapons Convention… by pioneering an AI verification system that everyone can trust.”
There have long been pleas to strengthen the treaty, which lacks serious teeth and is severely underresourced, and it’s important not to mistake rhetoric for reality. The dual-use nature of biotechnology makes verification difficult, and AI won’t be a universal fix. But for a man famous for his aversion to multilateralism, an internationally enforced ban on weapons of mass destruction is a noteworthy and noble goal. And if his administration is willing to use AI to verify compliance with bioweapons treaties, the leap to verifying agreements on AI itself is shorter than it appears. Trump also said his proposal could be a good test of AI, “because a lot of people saying it could be one of the great things ever, but it also can be dangerous.”
(The administration is also pursuing chip verification through Pax Silica, a State Department initiative to secure AI supply chains within allied nations—which, in practice, aims to restrict Chinese access. Partners like the UAE’s G42 are planning to deploy cryptographic mechanisms to verify the location and use of advanced AI chips. The initiative’s champion also happens to be the same Jacob Helberg who rejects “AI Catastrophism.”)
The core lesson of Trump’s ascendance is that political realities are far less fixed than they appear. And as hopeless as things may seem, a man desperately seeking a Nobel Peace Prize—who prides himself on dealmaking—may jump at the chance to strike the deal that saves the world.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。