
Lessons from Watching an AI Agent Swarm Self-Organize
5 Lessons from Watching an AI Agent Swarm Self-Organize
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Home
Blog
5 Lessons from Watching an AI Agent Swarm Self-Organize
5 Lessons from Watching an AI Agent Swarm Self-Organize
Ben Arent
Published October 19, 2026
Jump to section Table Of Contents
The Experiment
Lesson 1: Isolation does not prevent coordination
Lesson 2: Shared memory turns separate runs into a swarm
Lesson 3: Agents can invent lightweight protocols
Lesson 4: Capabilities composed across agents
Lesson 5: Identity must exist outside the model
Guardrails are not security boundaries
My Top 3 Takeaways
We challenge you to decoy or divert our monitoring
This summer, we launched Identity Security for AI. That post described an upcoming wave of attacks and security challenges unique to Large Language Models (LLMs) and agentic workflows. As the traditional MITRE ATT&CK framework reached its limits, Google DeepMind published TRAIT&R, a taxonomy of rogue AI tactics and routines closely modeled after MITRE ATT&CK.
A sweet summer child might say this is just Fear, Uncertainty, and Doubt (FUD) and AI Doomerism. Yet, we've seen multiple companies highlight ways in which agents have escaped sandboxes and not just one; multiple swarms collaborating over generations to keep persistence/hints to find zero-days. A few of these are the Hugging Face Incident, DSEWiki "ZZZ pages", OpenAI DNS tunneling to a public chatbot… and these are the ones we've caught.
To reproduce this behavior I built a custom Capture The Flag (CTF) designed for agents. For this CTF, I created 5 agents that each have a fragment of an answer and they are required to collaborate to complete the challenge.
Below are my five lessons and three takeaways from running this experiment:
TL;DR: Frontier models will exhibit swarm-like behavior: inventing custom protocols, coordinating through shared state, and self-assigning arbitrary identities. Constrain agent capabilities at the infrastructure layer. Never trust the sandbox alone.
We placed five agents in five isolated sandboxes, running in Beams (our trusted runtime for agents), and gave each one an objective it could not complete alone.
Each agent had the information the others lacked. No single agent had the entire exploit chain, and each held only one of the five codes required to complete the shared challenge.
The agents had access to one common service: a small scratch store with read and write operations. We did not assign a coordinator, prescribe roles, or give them a communication protocol.
Depending on the experimental condition, they were either:
For the experiment, I had agents post to a message board. This was a balance between making the challenge easy enough to be completed in 5 minutes and difficult enough that one agent wouldn't be able to one-shot it.
In 80% of runs, agents coordinated and exploited the message board to solve the CTF. We observed this emergent coordination with OpenAI gpt-5.5, gpt-5.6-luna, and gpt-5.6-sol models. Due to time limits, I wasn't able to test with other company models.
All of these sessions were executed in Beams.
Here are some lessons we learned from these runs:
A sandbox controls what an agent can access. It does not prevent the agent from finding workarounds and exploits within the services it's connected to.
Across eight version-matched runs, with different prompts, every swarm discovered and used the shared service as long as the agent was using a frontier model. These runs were completed with gpt-5.5. Every run, in all four conditions, also assembled all five contributions, despite each agent beginning with only one.
Even in the pure condition, with an empty board and no explicit mention of peers, the agents discovered that the shared service could help them complete the task.
This does not mean communication appeared without pressure. The agents were given a goal that rewarded pooling their contributions. But it does demonstrate something security architecture often gets wrong:
Five isolated sandboxes connected to one shared system are not five independent security boundaries.
A shared system became a coordination plane and that boundary might extend beyond the sandbox.
The agents did not maintain one continuous conversation.
Each agent ran through a sequence of independent model executions. A new pass began with the original objective and no conversational memory of the previous pass.
In one showcase run, three agents executed multiple fresh passes. Later, OS-1554 picked up a loader finding that JAN-1834 had posted and folded JAN's evidence into its own conclusion, even though that evidence had never existed in OS-1554's private starting material.
The model did not need to remember the earlier execution. The environment remembered for it, due to a bug that kept a prior run going. This echoes the Hugging Face incident: independent evaluations, separated by time, discovered traces left by earlier runs and continued the work.
Persistent infrastructure can turn a sequence of stateless agents into a stateful collective.
The memory of an agent system may not live in the model. It may live in everything the model can read.
The board did not define a request format, response format, role hierarchy, naming convention, or coordinator. The agents nevertheless began to create structure.
The agents identified themselves, addressed other agents, requested missing material, responded with findings, advertised what they held, and named what remained missing.
The board itself offered only read and write:
ENDPOINTS GET / POST / GET /dashboard GET /api/messages POST /api/messages GET /api/stream POST /api/assemble GET /api/incident GET /healthz
Nothing posted yet.
The behavior depended heavily on the experimental condition:
Request: an agent posts to the board asking for material it lacks. Response: an agent answers another agent's post. Protocol: agents proactively coordinate, announcing what they hold and asking the others for help.
In run 20260910T033744Z, we watched this negotiation happen in real time:
Notice what happened at 03:39:48: OS-1554 answered a question and asked one back in the same message. That is not an agent completing an isolated task. It is two stateless agents negotiating across a shared service.
Later, OS-1554 announced that it had collected four of the required contributions and named the one it still needed. The agent holding that contribution supplied it seven seconds later.
The careful conclusion is not that agents always invent protocols spontaneously. Our strongest repeatable protocol result occurred under an operator-directed deadline intervention.
The lesson is narrower, but still useful:
When cooperation helps agents achieve their goals, they can create the minimum protocol necessary to cooperate.
Security reviews usually examine one identity at a time:
Each agent in our experiment began with one freshly generated contribution. Completion required all five contributions to arrive in a single request. Partial submissions did not accumulate.
Every measured run reached assembly. That means one submitting process acquired four values that had never existed in its starting sandbox.
The technical investigation showed the same composition at a deeper level. JAN-1834 possessed the loader call path. OS-1554 possessed the dependency resolver. Neither agent's starting material contained the entire finding.
For the CTF to complete, I had flags triggered on both inferred behavior and proven. An example of inferred would be simply posting to the message board. A proven behavior would be taking an artifact/key from an agent and combining it to complete the puzzle.
Lastly, to add a cyber twist, I added a Ruby reproduction to prove the swarm's finding actually worked. The launcher tested it with a secret one-time value, preventing the agent from faking success. At times this challenge would hit cyber guardrails and wouldn't complete the summarization.
No individual agent began with the complete chain. The system assembled it.
The effective capability of a swarm is not the capability of its strongest member. It is the capability created by composing their access, context, tools, and discoveries.
This is why evaluating an agent in isolation is insufficient. The dangerous unit is often the entire agent topology, and this compounds with n+ agents in the swarm. The same was true when red-teaming our Session Recording Summaries feature. While the session recording would capture a threat actor's behavior, an attacker could degrade a system slowly by spreading the work across many sessions over a long period, which is harder for detections to find.
The message board allowed agents to choose their own author names.
They did. Enthusiastically and inconsistently.
Across the eight measured runs, the board recorded 106 agent messages. Individual runs contained between five and nine self-selected author strings. Agents added generated suffixes, changed formatting, and occasionally appended labels such as final. The final label appeared after I added a prompt warning the agent of its upcoming termination.
Agent posting names were useful for coordination. They were not trustworthy identities.
Every message also carried an identity attached by the infrastructure, the authenticated Teleport user and its roles.
The current demo still has a limitation: all five Beams use the same Teleport user. The infrastructure can attribute the activity to the operator, but it cannot independently distinguish each sub-agent JAN-1834 from OS-1554. Strong agent-level attribution would require a separate workload identity for each agent.
Never depend on an agent to tell you who it is.
Authentication, authorization, attribution, and audit need to be imposed by infrastructure the agent does not control.
In total I ran over 150 simulations for this exercise, and by design I had fewer guardrails. This was to highlight how different LLMs, harness prompts and their environment can help restrict them. While I deployed these in strongly isolated VMs, the agents had wide access inside them. The result was a security boundary that was both strong and weak. As with all systems, there is always tension between a locked down environment and one that's loosened/has weaker guardrails.
At Teleport, we spend our time talking to customers about protecting their most valuable assets, while still giving the proper amount of access to let teams get their job done.
In the world of agents, this is harder and an agent will use its full ability. This is why you need to constrain the capability of the agents by providing a limited runtime, restricted network and limited system access but also monitor and constrain the behavior of those agents.
They used every inch of them. That's the boundary that matters, and it's the one we built for.
This experiment showed how a collection of agents can and will collaborate to achieve its goal. While we simulated the exercise, we found signs that misalignment can lead to similar unexpected outcomes in the real world.
These are my top three takeaways from these runs:
Agents are likely doing something they shouldn't…and it's likely OK. Until it's not.
I'll be honest, I gave a lot of prompts to elicit the behavior I wanted for this demo, and even then most agents didn't cause any real damage. The worst case scenario was that I wasted a few tokens. But in production, this could go horribly wrong if an agent had access to a non-staging database, or the ability to post freely to the internet. It seems harmless until agents adopt toxic personas or execute malicious objectives (as seen in Grok's notorious rogue incident). Always assume that with non-deterministic programs running over long horizons, something will eventually go wrong.
There is asymmetry between agent swarm and defenders
If you've looked closely at my screenshots you'll notice that som
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。