
How worried should we be about extinction risk from AI?
A little over two months ago, Jacob Tsimerman surprised a lot of people by announcing, immediately after he had received a Fields Medal, that he would be giving up mathematical research and “pivoting to AI safety”. He h…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
A little over two months ago, Jacob Tsimerman surprised a lot of people by announcing, immediately after he had received a Fields Medal, that he would be giving up mathematical research and “pivoting to AI safety”. He had two reasons for this: that he did not see much future for the kind of research in mathematics that he liked doing, and that he was very worried about the dangers of AI, and in particular of the risk that it could kill us all, or x-risk as it is often known. (In this post I’ll use x-risk to refer to the extinction risk from AI rather than to extinction risk in general. I’ll also refer to “the probability of x-risk” when what I actually mean is the probability of extinction by AI.) Then in early September, he announced that he was setting up the Mathematical AI Safety Institute, or MAISI. He asked me if I would be prepared to be on the scientific advisory panel for this, and I agreed to do so, not because I am an expert, or because I felt as concerned as he did about x-risk, but because I don’t really know how much I ought to be worried about it. Given that, and given that the stakes are so high, I would like as many people as possible to be thinking seriously about it, and I agree with Jacob that mathematicians have the sort of training that makes them well suited to contribute. Jacob is the first to admit that something like an international treaty to slow down AI development is likely to have a far bigger impact on AI safety, especially in the short term, than mathematicians obtaining a better theoretical foundation for talking about it, but his view is that we need all hands on deck. If mathematicians can reduce the probability of x-risk by 0.1%, then surely that’s worth doing: the expected number of lives saved is over eight million.
Until recently, I have always found that my anxiety of choice was climate change rather than x-risk, but I was aware that there were a lot of very intelligent people who disagreed with me, and also aware that I had thought much less about x-risk than about climate change. So when I agreed to be on the scientific advisory panel, I also decided that that was the moment that I should think a bit harder about x-risk. Part of me had been actively resisting doing so, because I was worried that it would force me to face up to moral responsibilities that I would rather avoid: for example, if I came to believe that there was a 30% chance that AI would kill us all within five years, which in some circles is not an extreme position, would I be able to look my one-and-a-half-year-old grandson in the face if I wasn’t making a big effort to reduce that risk? I remember having a similar attitude a long time ago to reading Peter Singer’s book Animal Liberation after being told that you can’t read it and not become a vegetarian. I never did read it, and carried on eating meat, feeling only slightly uncomfortable about it. Eventually I did reduce my meat intake substantially, but that was because of my worries about climate change, though now that I have done that I find myself caring more about the animal-welfare aspect as well. (There is also a strange connection between animal welfare and x-risk, in that the nicer we are to animals, the easier it is to hope — not necessarily realistically — that a superhumanly intelligent AI might be nice to us.)
Anyhow, I’ve got past that phase now, and have started thinking about x-risk, reading up on the subject and trying to evaluate the arguments I read. My plan for this post is to explain where I’ve reached in that thought process, and possibly to clarify my thoughts as I do so. If you have got further than me, or are more expert than I am about x-risk and AI, which is not hard, then the post is not aimed at you. It’s aimed more at someone who is roughly at the stage that I was at a little while ago, in the hope that I might nudge a few people a little further towards taking the problem seriously. I will emphatically not be trying to persuade anybody that a particular position about x-risk is the right one — I am far too uncertain about it to be able to do that. But with all big issues like this, there are several layers to the arguments. The bottom layer consists of the arguments that jump to mind when you first start thinking about the issue. The next layer is common rejoinders to those arguments. Then the next layer is rejoinders to those rejoinders. And so on. I find it very disheartening to read a climate denier trotting out a layer-1 argument when its layer-2 rejoinder has been established for a long time and can be found after two seconds on Google (or, nowadays, an LLM). But until recently, my arguments against x-risk were basically layer-1 arguments, the sorts of arguments that would provoke eye rolls among the people who have thought properly about it. I hope that if you dismiss x-risk for layer-1 reasons, you will at the very least realize that there are answers to those reasons and you need answers to those answers.
I originally planned to publish this post just after the announcement of MAISI, but I hadn’t finished it, and then I became very busy and my progress stalled. During that hiatus, x-risk suddenly became a hot topic, and after a while I realized I would have to ditch what I had written and start again. For example, at the time I started it, I had been too lazy to read anything detailed about the Hugging Face incident, and so was unaware of some of its more disturbing aspects. And a couple of weeks ago I was sitting next to Jessica Wang, another mathematician who has turned towards AI-safety, at a conference dinner, and she gave me a copy of If Anyone Builds It, Everyone Dies, by Nate Soares and Eliezer Yudkowsky, a book I had resisted reading for Peter-Singerish reasons but was finally accepting that I would have to read if I wanted to write this post. I devoured it within 24 hours and found it better than I had expected (my expectations partly influenced by reviews I had read), and useful for organizing my thoughts. My original plan was for the post to be focused on a paper by Andrew Critch and Jacob Tsimerman that attempts to classify ways that AI might kill us all, giving examples of extinction scenarios in each class. But now, although I’ll discuss that paper, the post will be more of a reaction to Soares and Yudkowsky. It will also repeat many of their arguments. In fact, I may as well say from the outset that I make no claim that anything I write here is original. It’s hard to separate out what I have read from what I have come up with from thinking about what I have read, or thinking directly about the issues themselves, but even my own thoughts are likely to be standard thoughts that have been had by many other people.
This is turning into a rather long preamble, but I can’t resist telling a small anecdote that connects me to Eliezer Yudkowsky. In November 2024 he tweeted, “I would like to have a long recorded conversation with a well-known scientist who takes for granted that it is a Big Deal to ask if everyone on Earth including kids is about to die, who presses me to explain why it is that credible people like Hinton seem to believe that.” He had had such a conversation with Stephen Wolfram and found that while it had been valuable, it left him unsatisfied. Somebody suggested my name, and in the end it actually happened: Liv Boeree offered to host it, and we recorded a three-hour conversation. However, if you want to listen to it, you can’t, because it never appeared. I think that was probably because I didn’t take x-risk anything like as seriously as Yudkowsky, and more importantly I was offering little more than level-1 arguments of zero interest to him. Also, I had a few technical problems with the sound. Maybe if he reads this and wants to try again, I’ll be able to do a better job.
At the time of the conversation, which took place in January 2025, my main argument was something like the following. For AI to want to kill us all, it would need to have wants, but it’s not at all clear what it would even mean for AI to “want” something — it doesn’t really seem to care about anything. You can ask ChatGPT very rudely to carry out a task for you, and it just does so (or at least does its best to do so). If you switched it off, it wouldn’t care — even the concept of “caring” wouldn’t make sense. I didn’t dismiss the idea of AI ever wanting to do something, but I didn’t have a satisfactory account of what “wanting” would actually be, and didn’t see any sign of phenomena that deserved to be thought of as the kinds of wants that would be required for AI to be an existential threat.
I was sort of half aware that there was an answer to that argument, but I didn’t find it as convincing as I now do. The standard answer is that if you use a reward function to train an AI to have a certain goal, then you end up building it in such a way that it is very good at achieving that goal. If the goal is at all complicated, then to achieve it you will need to formulate subgoals, which may well not be what you expect. The initial goal can be thought of as a fundamental want, which spawns a whole hierarchy of subsidiary wants. Note that for this argument, it is completely irrelevant whether the AI has anything that we would want to call subjective experience. If you prefer to say that it acts as though it has wants instead of saying that it actually wants things, then be my guest: an AI that acts as though it wants to kill all humans is as dangerous as one that actually wants to kill all humans.
A famous version of this argument is Nick Bostrom’s thought experiment about an AI that is trained to create paperclips and ends up diverting all of the Earth’s resources to that goal. I think part of the reason I didn’t take the more general argument seriously enough was that I always found that paperclip-maximizing scenario too ridiculous: surely if the AI gets smart enough to divert the Earth’s resources towards making paperclips, despite our not wanting it to, it would at some point stop and think, “Hang on, why am I doing this?” To give an analogy, consider the very deep instinct that we humans have, to reproduce ourselves. There is no doubt that that goal, implanted in us by millions of years of evolution rather than a run of reinforcement learning, leads to exactly a hierarchy of subgoals as described above (for instance, it leads some people to try to earn a lot of money, and that in turn leads them to work at a job that they wouldn’t dream of doing if it were not well paid, often in order to earn a salary way beyond what they need to meet all their material needs, which earlier led them to study a certain subject at university, and so on and so forth). But despite our deep instincts, we also find ourselves capable of saying, “Actually, do I really want to have children?” and some people go as far as to take vows of celibacy.
However, even if a paperclip maximizer came to question whether it really cared so much about making paperclips, that doesn’t imply that the subsidiary goals it had developed would suddenly disappear as well. A key point that Soares and Yudkowsky stress over and over again is that the current AI paradigm is to grow models, for example by reinforcement learning, rather than carefully crafting them to do what we want. As a result, we have much less control over what they do, and in particular much less control over all those subgoals.
A closely related difficulty with AI (of the grown rather than crafted variety) is that we generally can’t reward it for actually doing what we want, but instead we have to reward it for achieving some proxy for what we want. For example, if we give it positive feedback every time it responds to a prompt in a way we like, it will be trained to respond to prompts in ways we like, even if to do so it tells us convincing lies, or confirms our prejudices, and so on. And it may do this even if we sincerely wanted it to tell the trut
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。