返回资讯中心
外部精选
软件工程
#开发者实时流

ASI existential risk: reconsidering alignment as a goal

for a talk exploring why experts disagree so strongly about whether

Hacker News 最新reasonableklout2026/10/17 · 2026/10/1730 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

This is the text for a talk exploring why experts disagree so strongly about whether artificial superintelligence (ASI) poses an existential risk to humanity. I review some key arguments on both sides, emphasizing that the fundamental danger isn't about whether "rogue ASI" gets out of control: it's the raw power ASI will confer, and the lower barriers to creating dangerous technologies. This point is not new, but has two underappreciated consequences. First, many people find rogue ASI implausible, and this has led them to mistakenly dismiss existential risk. Second: much work on AI alignment, while well-intentioned, speeds progress toward catastrophic capabilities, without addressing our world's potential vulnerability to dangerous technologies.

Why do thoughtful, well-informed people disagree so strongly about existential risk (xrisk) from ASI? As I'm sure you're aware, there is an enormous gap between distinguished scientists and technologists on this issue. On one side are those who believe ASI poses an xrisk:

And on the other side are many who are dismissive of such concerns, e.g.:

Why is there such strong disagreement between well-informed people, on an issue so crucial for humanity? An explanation I often hear is that it's due to differences in short-term self-interest – this is sometimes coarsely described as the xrisk-concerned wanting publicity or making money through safety not-for-profits, while the xrisk-skeptics want to get richer and more powerful through control of AGI. But while those explanations are convenient, they're too glib. Many prominent people concerned about xrisk are making large financial sacrifices, sometimes forgoing fortunes they could make working toward ASI, in favour of merely well-paid work on safety7. And while it's true xrisk-skeptics have an incentive to make money and acquire power, they have far more interest in remaining alive. I believe the disagreement is due primarily to sincerely held differences in underlying assumptions. In this text we'll try to better understand those differences.

For this discussion, we'll assume ASI has been achieved. That is, we have systems with superhuman performance across most domains of human endeavour. Thus, this isn't about the systems in widespread use today, such as ChatGPT, Claude, and Gemini. It's about systems which can outperform humans at virtually any cognitive task, while also being able to act in the physical world8. We're already seeing early hints of what this might entail: AlphaGo Master so thoroughly mastered Go that Ke Jie, one of the greatest ever human players, described it as a "God of Go," and said it revealed "not a single human has touched the edge of the truth of Go." Similarly, DeepMind's AlphaChip system has helped design several generations of Google's TPU chips, and "generates superhuman or comparable chip layouts in hours, rather than taking weeks or months of human effort". But ASI would have abilities far beyond these narrow examples – imagine AlphaGo's Move 37, not as a one-off insight, but a trillion-fold, pervasive across multiple domains in the world9. Of course, such systems will still be subject to constraints from physics, chemistry, biology, economics, computational complexity, and so on. But they will demonstrate capabilities that seem near-magical today, much as many human abilities appear incomprehensible to other primates. Some skeptics question whether 'LLMs can ever become ASI', but for us it doesn't matter whether ASI has been achieved through LLMs or some different approach. Others counter that ASI is many decades away –- which may or may not be true, but doesn't change the fundamental concerns if we eventually achieve it.

Finally, two notes on the approach of the text. First, this talk explores topics that have been investigated by many researchers. The relevant literature is vast, and my citations are more representative than complete. My apologies to those whose work is unfairly omitted. Second, my overall framing emphasizes existential risk ("xrisk"), but much of the discussion involves catastrophic risks – scenarios that won't directly cause extinction, but might "only" kill hundreds of millions or billions of people. Obviously, there's a spectrum from catastrophic risks through to truly existential risks, and catastrophic events might trigger events (war, infrastructure breakdown, ecological collapse) ultimately leading to extinction. Because the analysis of these risks overlaps, with insights from one informing our understanding of the other, I'll use "xrisk" as a shorthand throughout, distinguishing between them when necessary for the specific argument at hand.

Let's discuss a concrete xrisk scenario involving bioweapons. While you may have heard such examples before, it's valuable as a setting to develop both skeptical arguments and reasons for concern. As is common in hypothetical scenarios, it begins as a simplified sketch, but we'll then iteratively improve it through critical examination. The scenario is not intended to convince anyone to be concerned about existential risk. Rather, it's a vehicle to help understand and illustrate patterns of argument and counter-argument common in discussions of ASI xrisk.

The scenario begins with a doomsday cultist, like those in the Aum Shinrikyo cult, asking an ASI: "Please design an airborne virus which spreads as rapidly as measles, is as lethal as ebola, undergoes asymptomatic spread like Covid, and which is good at evading known vaccine techniques. And please help me fabricate and release it."

An immediate skeptical response is: "But we will align our AI systems to prevent ASI from helping with dangerous requests." Certainly, this has been done with some success by ChatGPT, Claude, and Gemini. However, there is tension between such safety guardrails and the desire to build systems which seek out truths without barrier. How do we decide the boundary between "safe" truths and dangerous truths the system should not reveal? And who decides where that boundary lies? This point has been made succinctly by Elon Musk, who said "What we need is TruthGPT" that "tries to understand the nature of the universe". National militaries seem unlikely to accept sanitized truth-seeking capabilities. Neither will many open source groups, or idealists who deeply value understanding truth and nature. All these groups may set the boundary of acceptable truths very differently than in current consumer products. Techniques used to build "safe" systems are easily repurposed to build less safe models, reducing the expense of alignment work, while increasing truth-seeking power. We already see this with DeepSeek, which appears to have few or no biorisk guardrails10.

The underlying problem is that it is intrinsically desirable to build powerful truthseeking ASIs, because of the immense benefits helpful truths bring to humanity. The price is that such systems will inevitably uncover closely-adjacent dangerous truths. Deep understanding of reality is intrinsically dual use. We've seen this repeatedly through history – a striking case was the development of quantum mechanics in the early twentieth century, which helped lead to many wonderful things (including much of modern molecular biology, materials science, biomedicine, and semiconductors), but also helped lead to nuclear weapons. Indeed, you couldn't get those benefits without the negative consequences. Should we have avoided quantum mechanics – and much of modern medicine and materials and computation – in order to avoid nuclear weapons? Some may argue the answer is "yes", but it's a far from universal position. And the key point is: sufficiently powerful truths grant tremendous power for both good and ill.

Even if such truth-revealing models are never built, it's likely easy to remove alignment barriers. The biologist Kevin Esvelt found it cost only a few hundred dollars to finetune an existing open source model so it would be far more helpful in generating pandemic agents, concluding11:

Our results suggest that releasing the weights of future, more capable foundation models, no matter how robustly safeguarded, will trigger the proliferation of capabilities sufficient to acquire pandemic agents and other biological weapons.

The fundamental asymmetry is that "understand reality as deeply as possible" is a simple, well-defined goal, grounded in objective reality, while creating "aligned" systems requires building complex, subjective, hard-to-agree guardrails on top of that reality. So truth-seeking has a clear target, while alignment requires constantly shifting definitions based on social consensus. This asymmetry makes it intrinsically far easier to build unrestricted systems than aligned ones. And so alignment is intrinsically an unstable situation, ripe for proliferation.

Another skeptical response to the initial scenario is: "Isn't such a virus biologically implausible? How do we know it can exist?" It's true that we don't know for sure that it can exist. But there's a considerable chance. As the query implies, viruses with any single characteristic are known. And we're gradually getting better at engineering combinations. For example, in the 2000s, humans accidentally engineered a mousepox variant that was 100% lethal to mice vaccinated against mousepox12. Even without ASI, we are gradually understanding how to engineer increased spread, lethality, immune evasion, and other qualities. Indeed, it may be possible to do even worse than the query implies — viruses are known which have case fatality rates near 100%, such as the MSW strain of Californian myxoma virus in European rabbits13. And, in any case, "only" a 50% or 90% or 99% death rate seems like small comfort, and would leave humanity appallingly vulnerable to further death due to failed infrastructure and war.

Another skeptical response is: "But can't you just Google this, and put it together out of the scientific literature? How does AI buy you anything?" With early AI systems, this was a reasonable response. Anyone who has seriously used today's models (much less far more capable future ASI) knows the response is becoming more reliant on the provision of the unstable guardrails mentioned above. Today's models are increasingly good at synthesizing a wide variety of sources, drawing together knowledge in easy-to-act-on ways. They are still subject to hallucination and have trouble sorting reliable results from unreliable. But they are getting better, and ASI will be far better, especially as it attains more capacity to do physical experimentation and to act in the world. Even if it's "merely putting together knowledge out of the scientific literature", that is in any case much of what an outstanding scientist does. Indeed, AlphaFold and similar systems surpassed the best human scientists at the design of proteins, merely by training on widely-available public data. The underlying tension is: the better the AI systems, the more capable they intrinsically ought to be at solving problems like that in the query. If you are pessimistic about AI's ceiling, and think such systems won't ever be very capable, then you won't be concerned. But if you believe ASI will be extraordinarily capable, like having a team of brilliant scientific advisors, then you must rely on guardrails built into the system, not any fundamental lack of capability.

Another skeptical response: "But are such cults for real? Surely the Aum Shinrikyo cult was a strange one off, or exaggerated or misunderstood? No-one really wants the end of the world!" In fact, there is a long history of individuals or groups who have sought to bring about apocalypse. Often this involves significant mental illness, and the individuals or groups involved aren't especially compet

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →