Kids Are Doomers Now

";inherits:false;initial-value:100%}@property --tw-inset-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-inset-shadow-color{syntax:"";inherits:false}@property --tw-inset-shadow-alpha{syntax:"";inherits:false;initial-value:100%}@property --tw-ring-color{syntax:"";inherits:false}@property --tw-ring-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-inset-ring-color{syntax:"";inherits:false}@property --tw-inset-ring-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-ring-inset{syntax:"";inherits:false}@property --tw-ring-offset-width{syntax:"";inherits:false;initial-value:0}@property --tw-ring-offset-color{syntax:"";inherits:false;initial-value:#fff}@property --tw-ring-offset-shadow{syntax:"";inherits:false;initial-value:0 0 #0000}@property --tw-outline-style{syntax:"";inherits:false;initial-value:solid}@property --tw-blur{syntax:"";inherits:false}@property --tw-brightness{syntax:"";inherits:false}@property --tw-contrast{syntax:"";inherits:false}@property --tw-grayscale{syntax:"";inherits:false}@property --tw-hue-rotate{syntax:"";inherits:false}@property --tw-invert{syntax:"";inherits:false}@property --tw-opacity{syntax:"";inherits:false}@property --tw-saturate{syntax:"";inherits:false}@property --tw-sepia{syntax:"";inherits:false}@property --tw-drop-shadow{syntax:"";inherits:false}@property --tw-drop-shadow-color{syntax:"";inherits:false}@property --tw-drop-shadow-alpha{syntax:"";inherits:false;initial-value:100%}@property --tw-drop-shadow-size{syntax:"";inherits:false}@property --tw-backdrop-blur{syntax:"";inherits:false}@property --tw-backdrop-brightness{syntax:"";inherits:false}@property --tw-backdrop-contrast{syntax:"";inherits:false}@property --tw-backdrop-grayscale{syntax:"";inherits:false}@property --tw-backdrop-hue-rotate{syntax:"";inherits:false}@property --tw-backdrop-invert{syntax:"";inherits:false}@property --tw-backdrop-opacity{syntax:"";inherits:false}@property --tw-backdrop-saturate{syntax:"";inherits:false}@property --tw-backdrop-sepia{syntax:"";inherits:false}@property --tw-duration{syntax:"";inherits:false}By Amol Kapoor

I was invited to speak to the Princeton AI Alignment group last weekend about the likelihood of existential risk and various forms of misalignment. I’m not fully sure why they invited me in particular, I’m just a guy on the internet, but I think the idea was that I have takes, and they wanted some takes, and the market cleared. Despite the rather dour subject matter, it was fun! Below are some miscellaneous things that we chatted about and some thoughts swirling in my head both during and after the chat, ideally more eloquently phrased than whatever I said at the actual meetup but less eloquently phrased than an actual coherent position piece. Thanks a ton Emmett for inviting me to come out.

It’s hard feeling 30 in a group of college undergraduates. It’s very tempting to make references to things like ‘remember how Facebook was in 2010,’ but doing that is always a mistake because you get hit with blank stares that just further underscore the age gap. What do you mean you don’t know what zynga is??? Still, I don’t envy the kids, they are coming of age in an extremely tumultuous period. It’s not just AI, of course. It’s…idk, everything? Have you looked around? This was the first time that I had the distinct sense that I had lived through a different era than the one we are in now. Like, just as an example, I was chatting with one of the students and it occurred to me that he simply did not remember a political cycle without Trump. He had never experienced a pro-social silicon valley,[1] or known a life without ubiquitous social access to everything. Imagine how much that all must change the perception of “normal.” It made me feel my age.

That was what we spent a lot of the conversation on — what does ‘normal’ even mean anymore? In some broad sense, misalignment is deviation from a shared norm. And even though the conversation started out about AI, it very quickly veered off topic to a wide range of other things, like the state of our politics, our economy, our culture. What are our norms in all of these domains? It was a bright and beautiful day out, the kind of day that only comes in October in the North East, and the kids were talking about their p(DOOM). That was the first thing they shared, actually. Everyone went around in a circle and ranked on a scale of 1 to 10 how concerned they were about AI risk. I think it was a log scale, which made the number of 7s and 8s and 9s surprising and concerning.[2]

I said my own level of concern was like a 5.[3] I normally am the doomer in the room, so it was an odd change of pace having to defend my position from the other side. If you want to read my more doomer takes, check here, here, and here.

The most clearly articulated concern was the standard Vernor Vinge / Tim Urban / Yudkowski mix of sci-fi terror and game theory, broadly summarized by one student as: “the AI will keep getting better, and you have written that alignment seems like an impossible problem, and there are other uncontrollable possibly malicious actors who are racing as fast as possible to unleash this on the world without any safeguards. So why are you not shitting your pants and freaking the fuck out?”

Honestly, it depends on the day. There are definitely moments where I am freaking out. I think the syllogism is straightforward enough.

But I’m also ten years older than I was ten years ago, and I think one of the things that has cropped up with that additional time is a sense that there are a lot more unknowns than we give credit for, and that the world has a funny way of being homeostatic. If the AI tools continue to get recursively better, and if we do not have sufficient time to develop monitoring and security tools, and if we are unable to coordinate some kind of response, then things may get pretty hairy. But at least right now, there needs to be a bit more ink behind those assumptions.

Take RSI, recursive self improvement. The basic idea behind RSI is that as the models get better, they will be able to continue finding more and better improvements in the process of training models, until AI begins improving itself at an exponential rate. In this model of the world, you could go from GPT-5 to GPT-50 overnight. RSI is a core assumption of the worst doomer scenarios, because it dramatically cuts our response time. Probably best explained by Tim Urban.

But this model of the world hides a lot of things. For example, it assumes that AI will get exponentially better based on linear increases in resources, even though everything we know thus far says the opposite. AI capabilities follow power laws, and models need exponentially more compute and exponentially more data in order to get linearly better.

It turns out it gets really hard to get exponentially more compute and data! You start running up against downstream things, like access to energy, and chip production capacity, and below that you run into mundane things like “where do you get your rare earth metals.” Humans are pretty damn smart, and humans with AI are even smarter, so maybe our recursively improving AIs find model architectures that are more efficient per watt, or discover ways of building chips that are easier to mass produce. But there are a limited number of such innovations, and as you start to pick low hanging fruit, progress slows down a lot. Even Moore’s law eventually ran out of fuel.

RSI assumes a positive feedback loop, but nature hates positive feedback loops. The only real example of an infinite positive feedback loop is a black hole. Every other positive feedback loop depends on fuel that runs out faster the faster things grow (nukes, population growth, etc). So in order to get a really rapid capability / intelligence explosion, the model’s rate of improvement needs to overcome the massive negative feedback loop that makes each incremental improvement significantly more difficult.

That’s not to say that AI won’t get better. But change is most disruptive when it happens very very quickly. I think we will have more time to adapt, at least more than the overnight capability explosions that would make a malicious AI really dangerous. In some sense, I think that’s the only question that matters — how much time do we have to figure out alignment at each stage of AI capability increase?[4]

Also, as an aside, it seems to me that the frontier labs are going broad instead of deep — in the last 6 months the models have definitely gotten better at writing code, but have gotten way way better at doing things mostly unrelated to writing code, like reading legal briefs. Some of this is downstream of the kinds of data that the labs are trying to procure. A model that knows how to do legal things probably can write code better than a model that can’t do legal things, just because of the bitter lesson and compression-as-intelligence. But some of this is because we are just running out of ways to make the models better at writing code!

Or take the “uncontrollable, possibly malicious actors” bit. Inevitably, every doomer has some variant of this conversation:

Layman: “just…don’t make the AI? Why is this hard?”

Doomer: “Well, we can’t just not make the AI, because if we don’t, other uncontrollable, possibly malicious actors will, and then they will <do bad things with it>.”

First, can we just appreciate how nuanced and unintuitive this argument is? I think a lot of the doomer crowd really really handwaves this part away, which makes sense because it’s easily the weakest part of the argument. It’s also worth noting that the doomer in question often works for one of the frontier labs, which, uh, really is not doing any favors. At best, it raises some questions about the doomer-ai-researcher’s general understanding of ethics (and hypocrisy).

In 2026, the term “malicious actors” basically always means the Chinese government, because there is no other geopolitical force that has the capacity to create frontier AI models. So the idea is that US labs cannot stop building frontier AI, because the Chinese labs will not stop building frontier AI. This is Anthropic’s favorite argument, which is why they mention China or the CCP 13 times in their statement on open source models.

ironically i mostly agree with the compute export restrictions but this really has very little to do with open models in the US

I always thought it was silly to assume the Chinese labs would not stop / be made to stop developing frontier AI if there was enough domestic pressure. You can make many claims about the CCP, but it would be really hard to argue that they don’t care a lot about stability. AI as a technology — with all the threats of mass layoffs and job replacement, not to mention the AI girlfriend/boyfriend industry — is profoundly destabilizing. So perhaps it shouldn’t be surprising that the Chinese government has already done more to regulate their AI industry than the US!

People seem to model China as a single monolithic entity that is solely focused on the destruction of the US, more like a terror group than a modern state that is responsible for over a billion people. Which is just a bit myopic, imo.

Either AI is massively destabilizing — in which case, the Chinese government will have very similar incentives to the US labs — or it isn’t. You can’t really argue that it will be destabilizing in the US but not in China (and if that’s the case, maybe we should just let the Chinese government build the AI)![5]

Ok but let’s say, just for the sake of argument, the Chinese labs don’t stop developing frontier models despite their destabilizing effects. And let’s also say that the US government does unilaterally pause model development. What then?

The maximally doomer position is hard for me to steelman, because I’m not sure I understand it completely myself. The Chinese government may