If the vulnpocalypse is the apocalypse, where are the agentic attacker hordes?

Joshua Saxe

Oct 23, 2026

66

12

8

Share

The Empire of Light — René Magritte

This piece took awhile to research with the help of Claude Opus 5.5 Max (and sometimes Sonnet when Anthropic downgraded me!) but it’s still a sketch of a more ambitious peer-review quality future version I hope to do with a collaborator. Feedback, as always, is welcome!

If you read X/Twitter or the tech press today, you might think PHP2Shell was a portent of impending doom. For those following closely, it was a predictable consequence of AI cyber capabilities improving on-trend, and also a positive update: defenders found it, along with tens of thousands of other serious security bugs over the past year, and it’s getting patched.

Attackers will of course exploit it. But like the other recent CVEs, the AI social-engineering tools that have been around for years, and the malware exploit kits that let unskilled attackers pursue lives of cybercrime, PHP2Shell is unlikely on its own to produce a significant bump in overall cybersecurity damages.

I show this in the plot below. Over the last decade, damages grew 3x as online scams, fraud, and ransomware industrialized. Notice two things: damages scaled rapidly before the launch of ChatGPT (and open-weights models like Llama 3) in 2022-3, and then actually tapered in the age of AI, 2022-now.

Why do the data contradict the vibes in the AI safety and security space? Why hasn’t a huge discontinuity in damages happened since the dawn of scaled AI? This non-responsiveness of damages to a rising tide of AI capabilities is the major conceptual crux, and stumbling block, of all AI security policy today. Everyone proposing an AI security policy intervention should have an explanation for the absence of significance uplift in damages from AI. Any theory of net new AI security risk, any theory of how to preclude or change those risks, and any proposed policy intervention needs to take stock of this lacuna.

We’ve ignored how cybercriminals are unskilled and get arrested a lot

Here’s the first part of my theory: cybercriminals are not the organized geniuses we implicitly imagine. Anyone building a criminal gang has to find candidates willing to forgo more stable opportunities in the formal economy. My modeling estimated that something like 720,000 people are involved in the online criminal economy, 90% of them in scam and fraud call-center type operations, some of them human-trafficked. Of the 76k or so actual hackers, the vast majority work for governments. Of the remaining few thousand criminal hackers, most work in ransomware. Some demographic estimates of these workforces are below.

Criminal organizations need to base themselves in middle and lower income countries, where they can pay less and hire people who have few formal-economy options. For scams and fraud, picture an illegal call center building in a small city in Nigeria. For ransomware gangs, picture kids working in the informal economy in Belarus. I think folks in the AI safety and cyber world sometimes imagine adversaries as their Waluigi twins, implicit MIT graduates; in fact, in cybercrime we’re dealing with a lower-skill demographic. The chart below estimates attacker wages relative to default economy wages.

Criminals also contend with the risk and reality of arrest, sanction, interdiction, and prosecution, which caps their ability to scale: the larger the organization, the more domestic and international law enforcement attention it attracts. Russia, Belarus, Colombia, and Ukraine have histories of weak rule of law, and Russia in particular has a record of looking the other way when attackers attack foreigners (e.g. in the US!). But when criminals scale to the point of attracting international attention, they’re often arrested on vacation abroad; the law does have a history of those who attract too much negative attention.

The chart below shows arrest outcomes for 152 named criminal enterprises (attackers, forums, botnets, and the like) I had Claude Opus look into (caveat: there’s obviously survivorship bias here). The large majority suffered legal consequences. All this is to say that outside of access to zero-day exploits, attackers face physical constraints, like law enforcement and access to talent, that defender organizations clearly don’t.

We’re underrating how long it will take cybercriminals to invent, operationalize, and scale new AI-centered business models

People seem to forget that gaining initial access is one of many steps in a cybercriminal’s workflow. Before it, they need to set up and maintain attack infrastructure. After it, they need to move laterally to get the data and access they need. In the case of ransomware (most criminal cyberattacks), they then need to ransom the victim’s data, negotiate payment, launder the money, and get it into their bank accounts, all while avoiding law enforcement, dealing with recruitment/retention/etc.

It’s not clear to me why the AI security and safety communities focus in such a distorted way on initial access costs (and sometimes privilege escalation and lateral movement costs) via new CVEs as the single bellweather for where cyber damages will go; these pretty clearly are not direct bellweathers.

It took the attacker ecosystem a very long time to mature the workflows, business models, and industrial divisions of labor that scaled ransomware into a multi-billion dollar industry, and it’ll take a long time to develop new models that centrally leverage AI throughout criminal labor processes.

Using AI to automate a few steps is one thing. What I mean by a new business model is fully automating the process, so that attackers can deploy armies of agents that handle most or all of the end-to-end work of extracting laundered cash from victims.

On top of the talent and law enforcement challenges above, attackers in the AI space have another problem: AI labor is purchased at a premium today, and reliable access to inference (from legal closed or open model providers) is clearly unstable. The chart below estimates how long it’s taken AI providers to kick attackers off their platforms. The point isn’t that providers do this quickly, reliably, or perfectly, but that these interdictions are, I’d imagine, an additional bottleneck on attackers trying to AI-automate their workflows. How would you like to build an AI-native company if the plug is constantly being pulled on inference?

We’ve also ignored how nation-states are slow to invent new operating models and adopt new private-sector technology

Governments have different offensive cyber workforces, motivations, and bottlenecks than cybercriminals. They have better access to talent and, often, to the capital to build expert cyberweapons and stand up AI inference. Still, a few bottlenecks help explain why we haven’t seen a step-function increase in offense where we might expect it (large-scale cyber-physical attacks from Russia to Ukraine, from Iran to the US/Israel, or the reverse).

For one, governments are historically slow to embrace and operationalize private-sector technology. See this chart on how quickly the US government adopted mobile phones, 2010s-era machine learning, and even remote video calls! I see no reason the US government’s offensive cyber units would fully retool around agentic automation quickly, even if they’re already using AI piecemeal, for instance to find vulnerabilities and develop exploits.

Governments also often have an access-to-tech-talent problem. The chart below estimates how much less governments pay AI cybersecurity talent than the private sector does among top-tier cyber powers. The incentive to leave for the private sector is strong, as many ex-NSA, ex-8200, and ex-GCHQ folks well know.

The other bottleneck is that fully automated cyber attacks, like agent swarms severely damaging a victim nation’s economy or critical infrastructure, will very likely rise to the threshold of physical warfare, so attacking countries will often be deterred by fear of retribution. The perhaps obvious but key point is that an increasing supply of cyber weapons, like an increasing supply of ballistic missiles, doesn’t necessarily mean an increasing volume of damage between competing states.

We haven’t seen a significant counterfactual contribution from AI to net global cybersecurity damages since 2022 / ChatGPT. But that doesn’t mean AI won’t transform cybercrime and cyber conflict and eventually lead to catastrophic damages. The apocalypse, I think, will be slow-moving.

I read a book about “cryptoviruses” in 2008, but it took 5-8 years for ransomware to become a huge deal. And ransomware eventually did hundreds of billions in damages, thanks to geopolitical conditions that let gangs sufficiently avoid law enforcement, a global political economy that pushed young people in the former USSR toward crime over formal software jobs, the rise of crypto payment rails, and the like. Likewise, “Cyber Pearl Harbor” was popularized in 2012, but it wasn’t until 2017 (NotPetya, Russia against Ukraine) that we saw a nation-state attack that significantly disrupted another national economy.

It’s our job, those of us in AI safety, AI security, and security, to anticipate bad cases, work out possible futures today, and start preventing them. The cases I’m worried about: nation-states developing agent-swarm capabilities that do 10-100x the damage of previous nation-state attacks; worms released by ideologically motivated actors that self-replicate, damage Internet availability, and cause enormous downstream harm; and new criminal innovation that substantially automates criminal workflows.

We should be crying from the rooftops that we’re in the calm before many potential storms, and that the time to prepare is now.