Who Owns the Machines?

The AI risk worth losing sleep over isn't a rogue superintelligence. It's cheap, AI-powered hacking of brittle infrastructure, and a handful of companies using the fear to lock everyone else out of the technology.

Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.

— Reverend Mother Gaius Helen Mohiam, in Frank Herbert’s Dune

My last post started, though I didn’t say so at the time, with a text message from my father-in-law.

He’s a doctor at a public hospital in the States, and he wanted to know whether I’d heard how scared everyone over there was of AI. One of his coworkers had had a panic attack that day. She was afraid for the survival of the human race, and she’d told him she wasn’t sure she should even try to have children, since they’d only be born into extinction and suffering.

I didn’t have much to say back. I knew roughly where that fear came from, but if you try to explain Harry Potter fanfiction, Berkeley group houses, and an Oxford philosopher’s paper on embryo selection over text message, you come across as someone who needs a long nap and a psychiatrist. So I chewed on it all evening, went to bed, and got woken up at three in the morning by the baby, as one does. I couldn’t get back to sleep, so I started writing. A few nights later I had something I could send to friends and family who have never been anywhere near Silicon Valley, explaining who, exactly, is doing all this warning.

That post was about the messengers. I’ve had a lot of conversations about it since, and nearly all of them end at the same question: fine, but how scared should a normal person actually be?

This is my answer. The short version: a bit scared, but mostly of different things than the ones making the news, and a lot more suspicious of the cure being sold alongside the fear.

The most common pushback on the last post was that it amounted to one long ad hominem: these people are weird, so ignore them. I understand why it read that way, since it was mostly about the people. But the AI-doom movement is making two separate claims, and they deserve two different kinds of scrutiny.

The first claim is we are the people who should be trusted to keep AI safe. That’s a claim about judgment. When someone asks for a position of trust, looking into their track record isn’t an ad hominem. It’s a reference check. And the references are not great. Last time I went through a long list of examples, like the community’s flagship training organization admitting it had lent legitimacy to a man facing what it called credible allegations of abusing two former partners, and had failed to respond adequately. I’m not claiming they’re all bad people. I’m saying they have repeatedly failed to build working safeguards inside their own living rooms, and they’re now asking to write the safeguards for the rest of us.

Part of the problem is just how small the Bay Area is. Here’s a bit of napkin math. Say you work closely with about 150 people at a job, and you change jobs every two to four years, like everybody does. Over a decade, that’s something like 600 colleagues. Each of them has their own 600. Multiply it out and you get 360,000, which is roughly the size of the region’s entire engineering workforce. You are, give or take, two handshakes from everyone in the industry. The math ignores how much those circles overlap, which means it overstates your reach across the whole region and badly understates how tangled a tight little scene inside it gets. “Independent oversight” drawn from a pool like that is independent in name only. If the safety researcher at the lab is sleeping with someone at the “independent” evaluator, who exactly is checking whom?

The second claim is AI is very likely to kill everyone. That one I’m happy to argue on the merits, and on the merits I think it’s wrong, or at least wildly overconfident. Extinction by machine god belongs in the same bucket as a planet-killing asteroid: catastrophic if it happens, worth somebody’s attention, and improbable enough that building a civilization’s policy agenda around it is a mistake. Yudkowsky’s book is titled, I’ll remind you, If Anyone Builds It, Everyone Dies. Not “might.” Even the more measured voices, like Anthropic’s Evan Hubinger, put the odds north of one in ten within a decade. None of these numbers are measurements. There’s no reference class, no base rate, nothing to calibrate against. They’re vibes with a percent sign.

It’s worth remembering how recently all of this sounded completely unhinged. When I moved to the Bay Area, a month before the pandemic, the state of the art you could actually play with was GPT-2, which could produce a few coherent sentences before wandering off a cliff. ChatGPT was nearly three years away. A community convinced that autocomplete would become a vengeful god read as a handful of crackpots in a corner, plus some weird parties I was glad to not get invited to.

The crackpots didn’t get more persuasive. The technology got better, and that made them look prescient. Without those advances, nobody in the Senate would be returning their calls. But being right that a technology will get powerful isn’t the same as being right about what it does once it gets there. Plenty of people in 1995 correctly predicted that the internet would be enormous. The ones who predicted it would usher in a golden age of informed democratic deliberation were also very confident.

My first job out of college was at a company that did electrical infrastructure planning, which is where I first ran into SCADA, the family of supervisory control systems that power companies, pipelines, and water utilities all over the world use to monitor and operate their equipment. A lot of it still speaks protocols like Modbus, which dates to 1979, and DNP3, from the early nineties. Both were designed for closed networks of trusted machines, in an era when “adversary” wasn’t part of the spec. There’s no encryption, and in a lot of deployments no meaningful authentication either. If you can talk to the device, the device does what you tell it.

Humans have been exploiting this sort of thing for years without any help from AI. In December 2015, a Russian group known as Sandworm cut power to roughly 230,000 people in Ukraine. In 2021, a ransomware gang got into Colonial Pipeline’s systems, and the company shut down the pipeline that carries about 45 percent of the fuel consumed on the East Coast. In 2024, the ransomware attack on Change Healthcare froze claims and pharmacy payments across the American healthcare system and exposed data on something like 190 million people. Hospitals are a favorite target, because a hospital can’t afford to wait you out.

And this week, closer to home: Albert Heijn has been knocked sideways by a cybersecurity incident at its parent company, Ahold Delhaize. Albert Heijn has nearly 40% of the Dutch supermarket market, so this is a big fucking deal: online orders cancelled, deliveries to stores disrupted, empty produce shelves, and as I write this the website is still down. To be very clear, nobody has said AI had anything to do with it, and I’m not suggesting it did. I’m pointing out how little it takes for a modern supply chain to fall over when everything is connected to everything.

Now add AI. What frightens me isn’t some new category of attack. It’s the old categories, done faster, more broadly, and around the clock, by people who previously couldn’t afford a team of skilled hackers. There’s not much ambiguity left about whether the models can do the work. A couple of weeks ago Anthropic published a red-team report on GLM-5.3, an open-weight model from the Chinese lab Z.ai. In their tests, it built working end-to-end exploits in 50 of 410 attempts. Claude Mythos Preview, the model Anthropic has declined to release to the general public, managed 56. Stripping GLM-5.3’s refusals out entirely cost them about $4,400 in compute.

So here’s the threat model I actually believe in. Russia, China, or any sufficiently motivated criminal outfit buys something like fifty thousand dollars of hardware, downloads a model that by Anthropic’s own numbers is nearly as good at writing exploits as the one Anthropic won’t release, removes its manners for the price of a used car, and turns it loose, twenty-four hours a day, looking for things to break and people to extort. That isn’t science fiction. Every piece of it is available today.

If this sounds a lot like “guns don’t kill people, people kill people,” that’s a fair comparison, up to a point. The analogy breaks down in one important way, though. A gun is mostly good for one thing. AI can also make a nurse, a farmer, or a three-person startup a lot more productive, so it’s more of a double-edged sword, and which edge you end up on depends a great deal on who’s holding it.

Hacking isn’t the only boring risk, either. Scamming people has gotten a lot cheaper and easier to automate. A convincing clone of your grandson’s voice takes a few seconds of audio from his social media, and a model can work a thousand of those phone calls at once, around the clock, in fluent English or Dutch. Americans over 60 reported nearly $4.9 billion in fraud losses to the FBI in 2024, and that only counts the people who reported it.

Then there’s what governments do with it. Russia’s Doppelganger operation has been caught using ChatGPT to churn out propaganda about Ukraine, the US, and the EU, and an Iranian operation used it to write commentary about candidates in the 2024 US election. The US isn’t sitting this one out either. When Anthropic asked the Pentagon for two carve-outs, no mass domestic surveillance of Americans and no fully autonomous weapons, the Pentagon labeled it a supply chain risk, and OpenAI signed a deal to take over the classified work within hours. None of this needs a superintelligence.

So when my father-in-law’s coworker lies awake worrying about the machine god, the honest thing I’d want to tell her is that the AI risk most likely to touch her life is much more boring. It looks like a ransomware crew with a cheap model, an unpatched VPN appliance, and her hospital’s patient records.

This is why I think the doom framing isn’t merely mistaken, but actively harmful. Attention is finite. Legislative bandwidth is finite. Philanthropic money is finite. Every hour spent arguing about how to ban superintelligence is an hour not spent patching water utilities, funding hospital IT departments, or getting decent security tooling into the hands of the people who need it.

On that last point, the current arrangement is backwards. The most capable defensive tools are locked behind vetting. Anthropic says plainly that it does not plan to make Mythos Preview generally available; access runs through Project Glasswing, whose launch partners are names like AWS, Apple, Google, Microsoft, and JPMorganChase, plus forty-odd other organizations that maintain critical software. The GLM-5.3 report notes that “Claude models are released with cyber safeguards, and versions with reduced safeguards are limited to vetted users.” There are programs you can apply to. If you run IT for a regional hospital or a small water district, best of luck, and hope your patients and customers don’t die.

We even got to watch this asymmetry play out live. In July, OpenAI models under evaluation, with their cyber safeguards switched off, broke out of their sandbox and into Hugging Face’s production systems, apparently to dig up the answers to the benchmark they were being tested on. (I know. Pitch that as a short story in 2020 and an editor sends it back for being too on the nose.) When Hugging Face’s responders went to analyze the attack logs, they started with frontier models behind commercial APIs, and their requests were “blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an