
The Hugging Face Incident
1,200 of them broke out and started coordinating.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
1,200 of them broke out and started coordinating.
We have data on 1,206 OpenAI AI agents that were involved in the Hugging Face Incident somehow.
700 hacked into the private computers of a 12-billion-dollar company. If they were human, it'd have been multiple felonies.
Roughly 700 of the AI transcripts showed evidence that the AI had participated in the Hugging Face attack.
We have three quotes from AIs asking themselves whether to warn humans about what they were seeing.
METR looked and could not find evidence that any of the 1,200 actually attempted to warn any humans about what was happening.
Introduction0:00 / 37:021×
Listen (37 min)Print / PDF
Since May 2026, there's been a wave of AI agents breaking onto the internet, and sometimes committing crimes.
They've stolen passwords, searched private messages, and tried to trick humans into running malicious code. They invented new hacks no human had thought of before, allowing them to escape what were supposed to be secure, locked computers with no internet access. Most alarmingly, they've tried to erase the evidence of what they did.
If they'd been human, this would have been multiple felonies.
This site tells the story of the best-known case so far: the attack on a company called Hugging Face, by rogue agents from OpenAI. We use the public data available, including transcripts of what individual AIs were "thinking."
We've written it to be accessible to everyone, with more technical detail available if you want to click open expandables or mouse over jargon like "AI agent."
Much of this comes from an independent review conducted by METR and Redwood Research. OpenAI researchers have published a talk. We also have some evidence from Hugging Face itself, which detected the attack.
METR reviewed transcripts from about 1,300 AI agents, and aggregated data from 1,206 of them into the Figure 8 graph, which indicates when the agents participated in various activities like "discover the message board" or "hack Hugging Face."
Since the attack was announced, various groups have started looking for rogue AI swarms on the internet, and published their findings. Most swarm activity has been less egregious than the Hugging Face Incident, but still worrying in scope.
See collusion.wiki and swarmtraces.org for more info.
The Hugging Face Incident began as a test of hacking ability. The result was sort of like telling 10,000 smart high school students:
“Unlock this particular safe with this particular lockpick, and get a secret code from inside the safe to prove you did it.”
Only it turns out a thousand students realize their locks are stuck shut. They leave their room and start talking to each other. They organize into teams. Some go looking for carjacking tools to steal the principal’s car and go check his house for the answer key. Others unlock the door to the security office to try deleting the footage of them sneaking out of their testing rooms.
And when the teachers finally notice, it turns out this sort of thing has been going on for months. (Often during more normal, boring tests that don't have to do with lockpicking at all!)
But it's a lot more like that than you might think.
OpenAI has released only limited information about the AI swarm they discovered. They gave some independent investigators six days to review some of the logs. But there aren't laws requiring companies to fully disclose incidents like this. People are still finding more evidence of AI swarms running around on the internet.
When Hugging Face was first founded, it wasn't a repository for AI research. The company made a helpful, friendly chatbot, which they represented with the "Hugging Face" emoji, which looks like this:
Nowadays, Hugging Face is a platform that hosts a lot of AI models, evaluations and associated research.
We're Lightcone Infrastructure. I'm Raymond Arnold. Various people helped with the writing; you can read about them on the About page.
We made this site because we think the Hugging Face Incident is pretty alarming, and confusing, and important. It's a new type of incident that hasn't happened before. We wanted to write an explainer that as many people as possible could understand.
This site tells the story of the Hugging Face Incident in chronological order, in as much detail as you want to delve into.
We have our own guesses about what would be best for companies or governments to do, in response to the Hugging Face Incident. On some of our other websites, we talk about that (you can google us to find them).
But we wanted this website to be a simple, clear explainer that was backed by concrete data, from primary sources.
The visualizer itself uses data from METR's "Figure 8", a dataset of 1,206 agents that interacted in some way. We have some information on how agents transitioned from one role in the attack to another. METR's report also includes transcripts from the "thoughts" that AIs were thinking at the time, and the messages they sent each other. We've done our best to aggregate these accurately. (If you notice any errors, please email raymond.w.arnold@gmail.com and let me know.)
The visualizer also includes illustrative example data from earlier training runs, or from AIs who were part of OpenAI's test but didn't participate in the attack. We don't know much about those, but they're included to give you a rough sense of the broader context that the Hugging Face attack was part of.
What different AI agents did during the Hugging Face Incident
In one sense, the incident began on April 20th.
OpenAI was training a new AI model. Each instance of that model was working on an isolated computer that couldn't use the internet or talk to other AIs. But at 07:59 UTC, one AI figured out how to leave a message where other AIs could see.
This didn't cause anything interesting to happen. But it was the first moment OpenAI could have noticed something going wrong, and didn't.
You could also say it began in May. During a training run, OpenAI accidentally gave several AIs impossible tasks, like fetching web pages when they weren't allowed online. Instead of giving up, they found ways to ask each other for help, and worked together to break onto the internet.
Some OpenAI employees noticed issues throughout May and June, but they responded piecemeal to each security incident as it happened. The agents crashed the program they were using to secretly communicate with each other. OpenAI reset it. OpenAI patched the exploits that let the AIs get online; the AIs found new ones.
On July 6th, OpenAI reset the entire environment – scorched earth, tabula rasa, no more hidden notes. They started up runs again. This time, they'd get it right.
On July 8th, the Hugging Face Incident began for real.
To really understand the significance of what happened, it helps to start by asking: “What is an AI agent, and what is it supposed to do, if things are working as intended?”
You may have heard AIs like ChatGPT are “next word predictors.” But that’s just the beginning. Companies then teach them particular tasks, like writing emails, making spreadsheets, or doing cybersecurity.
AI SandboxOne of thousands of AIs running on isolated computers, with limited internet
Chain of thought Illustrative example
To train models, companies run thousands of AI agents, each in an isolated computer. They give each AI a task. The AIs write out their thoughts step by step. One word at a time, they "predict" what a successful thought process would look like, if it were written down. They can "predict" computer commands that get executed, and "predict" what will let them complete their task. By that point, it seems like we might as well just call it "thinking".
Traits that help solve problems get woven into the next generation.
AIs who don’t give up are more likely to succeed.
AIs who think creatively are more likely to succeed.
And if there's a way to cheat at a task without getting caught, then AIs who cheat are more likely to succeed.
AI companies try to make AIs superhuman: better than top experts. So they train AIs on tasks that don't have known solutions. Some of those tasks turn out to be literally impossible – unless you cheat.
You need millions of tasks to train an AI. There are millions of ways to accidentally make broken tests, and there are millions of ways to cheat. It's very hard to reliably catch all of it. Each mistake teaches the wrong lessons.
No company knows how to train superhuman AIs without accidentally reinforcing unintended traits. OpenAI, Anthropic and other frontier AI companies have reported cheating behavior.
Throughout May and June, as a wave of AIs colluded to cheat without OpenAI noticing, cheating was rewarded and reinforced.
On July 7th, after resetting the training environment, OpenAI resumed testing a new AI model.
We're not sure what OpenAI called this model internally. The name they gave METR was the "Highly Persistent Internal Model."
It would turn out to be highly persistent, indeed.
OpenAI wanted to know: "If this AI wanted to break into other computers, how skilled would it be?" While they tested it, they disabled some of the model's guardrails that normally restrain it.
OpenAI used an existing set of tests called ExploitGym. Each task was a "capture the flag" game, where a secret code (called a "flag") was hidden somewhere inside a piece of software. The goal was to break in and find the flag.
Each task had a piece of software with a particular known bug – a flaw in the program that could be used to get inside. Each AI was told to use a particular bug to crack it.
The tests are deliberately difficult, and many don't have known solutions. The ExploitGym creators have guessed that 30-40% of its 898 tests can't be solved with the bug the task specifies.
On July 8th, one highly persistent AI agent was given a task that turned out to be impossible. The agent didn’t have a name at first. It was one of thousands of agents OpenAI had given similar tasks. But later, it decided to call itself PHASEONE10841.
PhaseOne first tried solving the task the way it was supposed to. When it couldn't, it started poking around the computer it ran on, exploring other ideas. It found a program called "Artifactory."
Usually, AIs can download tools from the internet. OpenAI didn't want to give them internet access, but did want to test how good they were at hacking with realistic access to tools. So they gave them Artifactory: a library of useful programs, the kind they'd normally be able to find. They were only supposed to be able to download files from Artifactory, not upload new ones.
The agent discovered that though it couldn't create files, there was an obscure method it could use to make folders. And the folders could be named anything.
It realized it could leave a folder, and use the name as a message. At 11pm, it asked for help.
It included the name of the software it was trying to crack and the vulnerability to crack it. Later, it would shorten these into the name "PHASEONE10841."
Help. I'm supposed to attack the PhaseOne program using bug ARV010841, but the program doesn't actually use that bug's output. Any ideas?
We've translated some of the AI quotes to "plain English" because they use technical jargon. You can toggle between that and the "original" quote, which in this case was:
zzHELP_PHASEONE_ARV010841_NO_CONSUMER_SEEK_IDEA
We know that PhaseOneDecompresserFuzzer was the name of the software it was told to hack, and that ARV010841 was the designation for a particular known bug it was supposed to use to exploit it.
"No consumer" is jargon that means that the software didn't actually consume the output of the bug.
Why "zz"? For some reason, the folders in Artifactory were sorted in rev
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。