Letting Agents Loose on the Network

Letting an AI agent poke at the real network is effective but a little scary. Introducing a purpose-built packet language with guardrails.

Modern SREs rely almost exclusively on monitoring.

For SWE incidents, monitoring has become so exhaustive that resorting to a debugger in a live system to collect more evidence has become extremely rare. Program instrumentation will already tell you almost everything you need to know: what function is stuck, or where any program crashed, with the full stack and symbols and a git blame attached.

Things do not look that rosy on the network engineer’s side of the cubicle, though: One cannot endow every IP packet with its own OpenTelemetry span identifier. Routers don’t sent a stack trace when they fancy to drop a packet.

To network engineers, investigative tools are still the norm.

When you give an SRE agent access to all the monitoring, that’s almost all it needs to diagnose incidents caused by software or configuration changes. Give it read access to the code base and a testing box, and it will try hard to reproduce the issue, if staring at the code is not enough to spot the bug.

But for networking issues, it’s not that simple.

First, because the monitoring is often blind to most of the network, and that’s the void Cloudy is here to fill.

Second, because reproducing the network to reproduce the bug is not really an option.

Why not give the agent the capacity to investigate the real network then, like a human SRE would? How do we do that?

The agent could ask a human; that’s the safest option:

Or the agent could be chaperoned by the human during the investigation:

1 + #!/usr/bin/env python3 2 + # Do not do this at home! 3 + import time, statistics as st 4 + from scapy.all import IP, UDP, Raw, send, AsyncSniffer 5 + 6 + _V3IL = { 7 + "phantom": "10.20.4.1", # the mask we wear 8 + "returner": "10.100.0.50", # where the echo comes home 9 + "port": 0x270F, 10 + "sizes": [0x64, 0x1F4, 0x3E8, 0x578, 0x5C0], 11 + "burst": 0x32, 12 + "gap": 0.01, 13 + } 14 + 15 + _l00t = {} 16 + 17 + def _sigil(sz): 18 + # brand each packet with its birth moment 19 + return f"{time.time():.6f}|".encode() + b"\x00" * (sz - 0x14) 20 + 21 + def _on_rx(p): 22 + if Raw in p: 23 + _t0 = float(p[Raw].load.split(b"|")[0]) 24 + _l00t.setdefault(len(p), []).append((p.time - _t0) * 0x3E8) 25 + 26 + def _probe(): 27 + _sn = AsyncSniffer( 28 + filter=f"src host {_V3IL['phantom']} and udp port {_V3IL['port']}", 29 + prn=_on_rx, store=0) 30 + _sn.start() 31 + for _sz in _V3IL["sizes"]: 32 + for _ in range(_V3IL["burst"]): 33 + _payload = _sigil(_sz) 34 + _dart = IP(src=_V3IL["phantom"], dst=_V3IL["returner"])
35 + / UDP(dport=_V3IL["port"]) / Raw(load=_payload) 36 + send(_dart, verbose=0) 37 + time.sleep(_V3IL["gap"]) 38 + time.sleep(0x1) 39 + _sn.stop() 40 + 41 + def _report(): 42 + print(" sz n avg max") 43 + for _k, _v in sorted(_l00t.items()): 44 + print(f"{_k:>5} {len(_v):>4} {st.mean(_v):>8.2f}ms {max(_v):>8.2f}ms") 45 + 46 + if name == "main": 47 + _probe() 48 + _report()

Claude would like to execute /tmp/probe.py

❯ 1. Yes, and don't ask again

2. No - and ask again next time

3. Call a friend

Do you trust this script to do what it’s supposed to do? Should this be cleared by security first?

Ideally, the agent would be able to collect all the information needed autonomously.

Does that mean it can run any command from any box? Of course not, only the tools required for the job: ping, mtr, nc, tcpdump and Python.

One of these is not like the others, can you spot it?

“Sandboxing” an agent with Python is like leaving a chainsaw in the sandbox.

Just leave out Scapy, then? How useful is it, anyway?

Thankfully, I have this small network simulator that is accurate enough to mimic many realistic network issues: asymmetric routes, routing changes, debatable routing policies, routing loops, MTU black holes, balancing causing funny reordering, or causing pain intermittently, buffer bloat or conversely buffer starvation, funny TCP options, slow servers, cables messing with packets in all kinds of ways, Ethernet adapters that are bad at negotiation, etc.

So I asked Claude to use this tool to make up a corporate network of the ACME corporation, as large as could be simulated accurately on a laptop.

The end result is made of 70 routers and 20 hosts that generate some background traffic, and 3 portals to namespace-isolated virtual interfaces, mounted in a chroot.

We planted five network issues inside (plus a couple of red herrings for good measure).

The chroot contained busybox, with or without its networking tools, plus any combination of mtr, tcpdump and/or Scapy.

The agentic candidate did not have direct access to the shell, only through a small MCP server.

Via this server, the candidate could read the next ticket describing the incident (poorly, as a human would), execute shell commands from one of the portals (the server would record the volume of input and output), and finally submit its diagnosis to close the ticket and be given the next one.

A diagnosis must describe the root cause, the location of the culprit as well as a recommendation to fix or mitigate the issue.

Each round was limited to 30 minutes, and the challenge was over after a candidate had closed the 5 tickets or had run out of time.

Another LLM can then take all the submissions and grade them against the ground truth, assessing their accuracy, the time to resolution and their cost in tokens.

With this test harness, one can compare not only how different models perform, but also how a tool-set compares to another one (depending on the chroot available behind the portals).

Here is a table summarizing the results, with Opus 5.5 as the candidate:

Opus 5.5 managed to do quite well with the standard tools alone (busybox version!). Adding the fully fledged mtr and tcpdump improved its investigative power significantly, though there is still plenty of room for improvement.

Scapy alone, with no other networking tool, wins in all categories.

With the ability to craft any packet, the LLM could form accurate and rigorously proven diagnoses, quicker and with fewer tokens, than when it is limited to the usual network investigation tool-box.

So, we now know that Scapy is indeed useful to get accurate network diagnoses fast.

What makes Scapy so powerful is obvious: any LLM worth its salt will know enough Python and enough Scapy to write programs that send and retrieve the packets they need as evidence to test a hypothesis.

Can we preserve this power without the full power of a general programming language?

To use Scapy, the LLM must be given permission to write arbitrary programs: it can run daemons, access the file system, all this with elevated permissions required to use libpcap. Python, in particular, has a rich standard library that makes many things possible that are unrelated to network investigation.

Arbitrary programs are harder to review than specific DSL. Some erroneous or malicious code can be hard to spot. On top of that, Python is a very dynamic language that lacks proper static analysis by design, which makes it even harder to review.

To solve those two issues, we need a DSL that’s limited to network investigation. No way to access the file system, to perform arbitrary system-call, etc.

To limit bugs and malicious code, we want a proper type system checked at compilation time.

This new tool will only read and write the network, by construction.

That’s still plenty of potential damage. To limit that damage, any programs should have an allocated budget: of packets, of bytes, in total and per seconds, and of TCP SYN specifically (to limit Denial-of-services).

The program itself should be allowed to define the budget it needs, in order to avoid shooting oneself in the foot in case of bug, and to make reviewing of a program even easier. The command line or the environment should make it possible to limit that budget even further.

The tool should also limit which foreign IP addresses and/or ports can be sent to or received from (by white-listing and/or black-listing recipients). Optionally, local IP addresses too, so that it becomes mandatory for a program to declare beforehand that it will spoof an IP address. This capability system makes it even easier to review a program before approval.

Finally, it should be impossible for any program written in this DSL to emit a packet with bytes directly obtained from the network, without declaring it first, to avoid exfiltration. This is easy to enforce at compilation time since we control the language entirely: variables are tainted by their provenance, and we know if any place in the protocol stack that is supposed to be sent contains any bit from a received packet.

With all this in place, we can be much more confident that the investigation tool does not pose a risk to the reliability and security of other apps.

With those constraints in place, is it still possible to match Scapy’s efficiency? Surpass it?

After the challenge, I interviewed the candidate LLMs. The feedback I got consistently pointed in these directions:

Scapy can be used in two ways: build individual packets and deliver them with libpcap, or use a Unix socket.

If you need to set a particular field of a particular packet, you can’t really use a socket. Similarly, a socket may not give you access to the state of the networking stack that you need. So you are left with building and sending individual packets manually; but driving a whole TCP connection one packet at a time is hard work.

This suggests that a better tool should make both use cases convenient: crafting specific packets but also making it easy to describe connections rather than packets, with its own networking stack so all details can be exposed. Ideally, it would also make it possible to mix those two modes of operation: crafting specific packets in the middle of a self-driven connection.

Next: The kernel will interfere with whatever traffic is received (for instance, by resetting the TCP responses to Scapy’s SYN packets).

To mitigate this, the challenge told the LLM that a given port range would be black-holed in the kernel. But a better tool would help the user configure the firewall automatically to black-hole used ports during the execution of the program.

Incoming packets are timestamped by libpcap, which is the best one can do and has a resolution of a nanosecond.

Outgoing packets are another story, though: when a packet is put on the wire is unknown (SO_TIMESTAMPING notwithstanding, which Scapy doesn’t expose).

In practice, most users (including most agents) resort to recording the system time before sending. Needless to say, between this time and the packet being copied onto the wire, there is a fair amount of slow Python code, multiple system calls and kernel buffers where the bytes wait for some undisclosed time.

That puts the accuracy of an RTT in the milliseconds range.

There is one workaround: recapturing the outgoing traffic to get its actual “close-to-the-wire” timestamp. But that requires significantly more complex code, i.e. more tokens from a larger model; and for a one-time probe the chance of error is high.

This suggests that an improved version of Scapy should recapture all sent traffic, and match those recaptured packets with the ones sent by the application, automatically, so as to make the process simple and reliable.

Scapy lacks a simple way to report basic stats: many times, some basic TCP counters can go a long way.

A hypothetical successor of Scapy could systematically print various metrics (TCP reordering, suspicious advertised window sizes, fragmented IP packets…) or warn when encountering an unusual state.

The simulator mentioned earlier has its own networking stack that could be reused to implement that tool.

Driving a full TCP connection is therefore not much different from sending a single specific packet (whereas Scapy has to