Two RTX 5090 hosts sharing a KV cache over Ethernet

KV-cache sharing lets a coding-agent session move to a second GPU host without recomputing its prompt. On two RTX 5090 machines, how much does the link between them — 1 GbE versus 10 GbE — actually change that?

A large language model does not reread your whole conversation from scratch on every turn. As it works through the prompt, each layer produces key and value tensors for every token — the KV cache (also called the prompt cache). Keep that cache and the model can skip straight to generating the next token. Lose it and it has to push the whole prompt through the model again first, a step called prefill. On a long coding-agent conversation that can mean reading tens of thousands of tokens before the first new one appears — seconds to minutes of waiting.

The catch is that this cache lives in the GPU process that served the turn, on one machine. A viiwork mesh normally sends a session's next turn back to the same host for exactly that reason: its cache is still warm. But when that host is busy, the turn has to run somewhere else, and the new host starts cold — a full prefill before it can answer.

KV-cache sharing is viiwork's way around that. After a turn, kv_share copies a replica of the session's cache across the network to the host the session would spill to next. When the switch happens, the receiving host imports that replica instead of recomputing the prompt, and the turn resumes warm. The copy only helps if it reaches the other host before the next turn arrives — so the link between the two hosts matters.

This post asks how much it matters. I ran the same coding-agent workload on two RTX 5090 machines while changing the connection between them: 1 GbE through a LAN switch, a direct 10 GbE cable carrying Tailscale traffic, and that same cable with kv_share sent directly. Each host had four model slots. The agents built a todo app from a spec, and I measured both 12-session and 16-session runs over 90 minutes.

Why would a session move at all? Because a host can be busy when the next turn arrives. The preference to keep a conversation on the host with its warm cache is called session affinity; when that host is full the turn moves, and kv_share is what makes the move cheap. The companion session affinity study shows why keeping a session near its cache matters in the first place.

To see what the copy buys, picture the move without it. When a turn spills to a host that has never seen the conversation, that host has no cache to reuse: it runs the entire prompt through the model before it can emit a single new token. The cost scales with the prompt, and a coding agent's prompt grows all day. The slower the card and the longer the conversation, the worse it gets — a cold prefill is about 41 seconds for 50,000 tokens on an older RTX A4000, and minutes at 200,000 tokens on the gfx906 fleet. On fast RTX 5090s it is quicker, but it is still pure repeated work: the receiving host re-derives a KV cache that already existed on the host the session just left. Session affinity avoids this most of the time by keeping a session put, but a busy host forces the move, and without a shared cache every forced move pays that cold prefill.

The engine running the model here is Strata, which serves the Qwen3.8-Flash-Next mixture-of-experts model and holds each conversation in its own cache. kv_share is a Strata capability that viiwork drives, and it moves the cache in three steps:

The full mechanism, its config (group, min_tokens, max_gib) and its requirements are in the kv_share documentation. The save and the import happen on each host; the one step whose cost the network governs is the send in the middle — which is the part this experiment changes.

The chart uses the 16-session runs, where the load stayed busy and the phases are most comparable. A typical replica delta was about 247 MiB. At 1 GbE it took 3.24 seconds to send. Over the 10 GbE cable through Tailscale, it took 0.81 seconds. Bypassing Tailscale cut the send to 0.55 seconds.

The raw cable carried 9.41 Gb/s in a single-stream test; Tailscale carried about 3.0–3.4 Gb/s. That explains why the first jump was roughly fourfold rather than tenfold. The direct socket was quicker again, but by then reading, hashing, and writing the replica at the receiver appeared to set the pace.

Yes, modestly. In the 16-session comparison, warm host switches rose from 93.8% at direct 1 GbE to 96.8% over 10 GbE with Tailscale. Switches that reused some, but not enough, of the prompt fell from 5.4% to 2.2%. A faster copy was more likely to reach the other host before the next turn arrived.

The direct 10 GbE run sent replicas faster still, but its warm share was 96.2%. That difference from 96.8% is within the variation between these runs. At this load, removing Tailscale improved transfer time without a clear end-to-end cache benefit. Forcing the direct cable down to 1 GbE also reproduced the switched 1 GbE send time, so the LAN switch itself was not the bottleneck.

An import into the receiving engine still took about 2.8 seconds at every link speed. Cold switches stayed around 1–2% in the busier runs. Traces showed a recurring software problem: viiwork remembered that it had imported a replica, but the engine could later evict that conversation. On a return visit, viiwork skipped the import even though the engine no longer held the cache. That import-once bug was fixed in viiwork v2.11.1 by asking the engine what it still holds before skipping an import.

For these two hosts and this workload, 1 GbE was workable: average kv_share traffic was roughly 320–450 Mb/s in each direction, and most switches were warm. A 10 GbE link made transfers more timely and reduced partly warm turns. The larger remaining gains were in engine import and cache correctness, rather than more network bandwidth.

Method note: these are single 90-minute runs per configuration, not repeated statistical trials. The 12-session runs were less evenly loaded, so the comparison above uses the 16-session J, H, and L phases. A “warm” switch reused at least 90% of its prompt; “partly warm” reused 30–90%. The measurements were made on 26 October 2026 with viiwork v2.11.0-beta1 and Strata 0.2.6, before the import-once fix.