Why harnesses matter for energy (and token) efficiency

At Neuralwatt, we can see how the agent harness calling a model affects the energy it uses. I've been looking through a month of traffic on our cloud, and the harness seems to decide a surprising share of how much work our GPUs do. We usually talk about inference efficiency in terms of the model (how many active parameters, its attention mechanism) or the hardware (H200 vs B200 vs B300). The client deserves a closer look.

Almost all of our traffic is agentic, with far more input than output. A typical coding-agent turn sends 100k tokens or more of conversation and gets back a few hundred tokens. Most of that input is the previous turn's prompt with a little added at the end. If the earlier text is unchanged, the server already has its key-value (KV) cache and only has to process the new part. This is prompt caching, our biggest source of efficiency. Serving a cached token is far cheaper than computing it again.

Caching needs an exact prefix. A cache break happens when the harness changes something it already sent. We then recompute everything from the first difference onward. A change near the beginning, even a timestamp, means recomputing nearly the whole prompt. A change near the end costs little. The harness controls how the prompt is built.

Every request tells us which client sent it. In the 30 days from September 8 to October 7, 2026, we saw hundreds of different clients, including open-source coding agents such as OpenCode, Oh My Pi (omp) and Pi. The mix also included editor integrations, the OpenAI SDKs and custom apps. OpenCode alone is used by about one in four of our customers.

For the most widely used clients, here is the share of input tokens we could not serve from cache:

Every client here caches the large majority of its input. Those small differences still matter: a client at 97% cached sends us three uncached tokens per hundred; a client at 92% sends eight. That is 2.7 times as much prefill work for the same conversation.

Hit rate also depends on the workload, not only the harness. Longer sessions score higher, because each turn's new text is a smaller share of the prompt, and about half of deepseek-harness's lead comes from users who run very long sessions. A cleaner measure of the harness is how often a turn breaks the cache: about 1 turn in 100 for deepseek-harness, against about 3 in 100 for OpenCode and Pi. We see both OpenCode 1.x and 2.x in our traffic, so the bar combines them. 2.x looks better alone (2.2% uncached against 3.0%), mostly because of the model mix. On the same model, they cache about equally well. 2.x fixes most of the 1.x cache breaks we found in the lab. That difference doesn't show up in production, and we don't yet know why.

We bill on measured energy, so we can see the cost directly. Over the week of October 1 to 7, 2026, we compared requests on the same model with the same prompt size (about 50k tokens) and similar output length. Here is the extra energy each request used compared with a fully cached one:

A mostly uncached request takes roughly twice the energy of a cached one of the same size. On GLM-5.3 it is closer to two and a half times. Prefill also competes with generation for the same GPUs, so wasted prefill slows down everyone else's responses on that server.

We followed individual sessions turn by turn on our three busiest models for the week of October 6 to 13, 2026. We looked at every turn that landed on the same server within five minutes of the previous one, where the cache should have been warm. Most cache breaks were small: a few thousand tokens near the end of the prompt got recomputed.

The delay is noticeable too. Here is the median time to first token for a normal turn compared with a turn that had to re-read most of its prompt:

I pay more attention to how deep into a session the cache breaks than how often it breaks. Changing the beginning of the prompt once, early on, costs little. Do it at turn 200 of a long session and you get a large, visible pause and a large energy cost. Long agent sessions account for most of our tokens, so this adds up.

We can't always tell from the server side whether a harness edited the prompt or the cache was evicted under memory pressure. These numbers include both. Our lab tests below show that harnesses cause the early-prompt edits we could reproduce; eviction likely accounts for some of the rest.

I looked for the causes in two places: the lab, where we could see every token, and production, where we only see request metadata.

We ran four popular open-source agents (OpenCode, Pi, omp and Kilo Code) through a recording proxy against DeepSeek V4.1 Flash and GLM-5.3. We used the latest releases as of October 6, with default settings: OpenCode 1.18.34, Pi 1.0.4, omp 18.6.3 and Kilo Code 7.8.3. We then repeated the session on OpenCode 2.0.29 from the 2.x line. Each worked through the same scripted session on a small Python project: plan, then build, switch reasoning effort and back, spawn a sub-agent, sit idle, get restarted, connect and disconnect an MCP server, add a skill, and run long enough to compact. We compared every request with the one before it, both as JSON and as the exact tokens the server renders.

Of 539 requests that followed an earlier one in the same conversation, 88% were pure appends: the previous prompt was an exact prefix of the new one. The other 65 broke the cache. Every break came from the harness changing something it had already sent; none came from our chat templates. Idle time caused no breaks in any of the four. Neither did restarting the harness and resuming the session, or using sub-agents.

Two causes break the cache without any action from the user:

Pi and Kilo Code had no breaks of this kind.

The other causes depend on what the user does. How much they cost depends on where the prompt changes:

Switching back to a setting we'd used earlier in the session was cheaper than I expected. The server still holds that earlier version of the prompt. Switching effort back hit 10.6k to 27.5k cached tokens in our runs. A toggle costs one full re-read the first time each new state is used, not on every flip.

We don't log or keep the text of customers' prompts or responses. Everything in this section comes from request metadata. The token-level analysis above comes from our own scripted test sessions, not customer traffic. That metadata records the reasoning effort each request asked for and how many tools it sent. We used it to check two of these causes against real sessions from October 6 to 13, 2026, on DeepSeek V4.1 Flash, GLM-5.3 and GLM-5.3 Flash. We used consecutive turns of at least 8k tokens on the same server within five minutes. To keep a few heavy accounts from driving the result, we left out the three customers with the most re-read tokens in each group.

Changing the tool list or effort level makes a cache break more than 20 times as likely and costs, on average, about 40% of the previous prompt. Tool-list changes happen on about 0.5% of turns, yet account for 19% of all re-read tokens on these models. That share is concentrated: three accounts make up about two thirds of it. The per-turn effect holds across everyone else.

The remaining three quarters of re-read tokens come on turns where neither the effort nor the tool count changed. That "nothing we can see" bucket holds the causes we can't see from the server: the reminder in OpenCode 1.x, omp's pruning, skills, the date, compaction and cache eviction. Next, I'd like us to record where each prompt first differs from the previous one. That would let us split the bucket up without reading any prompt text.

In our lab capture, every one of deepseek-harness's follow-up requests was a pure append to the previous one, even after the process was restarted and the session resumed. Reading its source, I found a few practices worth looking at:

Everything in this post comes from data we record for every request, and you can see the same data for your own work. Session Insights in the Neuralwatt dashboard shows each agent session turn by turn: how much of each prompt came from cache, how much had to be computed cold, the time to first token, and the measured energy and cost. It never shows prompt or completion text.

This is one of my own sessions from September 26, 2026: 35 turns of a coding agent on DeepSeek V4.1 Flash, growing from 29k to 257k tokens of context. 95% of its prompt tokens came from cache, and the small red caps are the new text each turn added. A cache break shows up as a tall red bar, so if you switch reasoning effort or connect an MCP server halfway through a session, you can see what it cost. You can use it to compare harnesses or settings on your own workload, and to check that a change actually kept the cache warm.

We route each session back to the server that holds its cache. On our busiest models, that already keeps 88–96% of each session's tokens on its home server. When a session has to move, our shared KV cache pool lets the new server load the prefix instead of recomputing it.

We recently measured how much more a smarter prefix-aware router would recover on top of that: a tenth to two tenths of a percentage point of hit rate. Most of the remaining waste comes from how prompts are built.

Next, we plan to send fixes and suggested defaults upstream to the open-source harnesses our customers use most. We also want to research ways for the server to reuse cached work after a small edit near the beginning of the prompt, such as the effort line or the date. That would help with clients we can't change ourselves.

A cache break doesn't show up as an error. You still get your answer, just later and with more energy spent on it. Measuring the energy of every request showed us how much that waste adds up. Most of it turned out to be fixable with a setting or a small change to how a harness builds its prompt.

AI runs on energy. The more of it we can see, the more of it we can stop wasting, whether that's a single developer's session or the industry's next data center.

Methodology notes. We identify clients by the name in each request's User-Agent header. SDK and runtime labels (Node.js apps, OpenAI SDK apps) cover many different applications. We only show clients used by many customers and never identify individual customers. Servers cache prompts in fixed-size blocks, so even a pure append can leave a few tokens at the end of the previous turn to re-read. We count those as uncached input, adding under 0.1 percentage points for the coding agents in Figure 1. Cache rates are the share of prompt tokens served from cache across all models from September 8 to October 7, 2026. Energy is measured GPU energy apportioned to each request; the cached vs uncached comparison holds prompt size and output length fixed and covers October 1 to 7, 2026. The deep-session and cause analyses cover October 6 to 13, 2026, on DeepSeek V4.1 Flash, GLM-5.3 and GLM-5.3 Flash, following each session's consecutive turns on the same server within five minutes.

Chad Gibson is the co-founder and CEO of Neuralwatt. He spent more than 20 years at Microsoft before investing in early-stage startups and co-founding Neuralwatt in 2024.