Give your coding agent a verification harness to check its code

Verification is the bottleneck. Give the agent somewhere to run the code.

Addy Osmani's Agentic Verification makes the case in one line: "I would always prefer a much better feedback loop to a slightly better model." A coding agent that can only read the code guesses. One that can run the product and use its interfaces and APIs can check and iterate. This is what a CI pipeline is, but placed inside the agent's session, and running in shorter cycles. We call that a verification harness.

At Heal the harness did not start as a product. It started as a question our engineers put to Claude Code in most sessions: what do you need to test your code?

Five weeks of answers became a Docker mode in the backend, a replay mode in the agent, integration specs that combine both, and a bench that drives a real Vercel sandbox. This article is about building that harness: what it is, how we designed it, what it caught, what changes when the same model runs with and without one, and why Heal now automatically builds one for every codebase.

“I would always prefer a much better feedback loop to a slightly better model.”

A verification harness is three things, made available to a coding agent, so it can verify its code.

Heal's agent runs long tasks over LLMs and reports them to a backend, so many of its bugs showed up only on staging or in production. The stack already had broad test coverage, and it was not enough.

Over five weeks the engineers answered that with one habit: ask Claude Code what it needs to test its code, or say how to test it locally, and let it build a first harness version.

The harness is kept decoupled from the production code as much as possible. In our specific case, two parts of the harness could not be completely decoupled:

So we used two patterns to deal with this part of the system: dependency injection, and ports and adapters. This was applied to:

This is controlled by an env var, so the same binary runs in production and in a test, and a verification harness reaches into the production code without a fork of it. Locally, we can also toggle between the stubs and the real dependencies.

DOMAINPORTADAPTERS · AN ENV VAR PICKS ONEheal-agentthe run loopasks for the next turnport · the modelquery(session) → turnan interface, no vendorClaude · the real SDK and tools · every production runreplay · HEAL_REPLAY=<recording> · plays a recording, $0heal-backendthe run servicestarts a run's machineport · the boxstart(run) → a machinean interface, no vendorVercel · a sandbox per run · production (noop: the default)Docker · AGENT_RUN_BOX_PROVIDER=docker · a container · devproductionvalidation

Figure 3. The port-and-adapter pattern on Heal. An env var controls what gets called.

3. What the verification harness caught

Let's take a look at six examples the harness caught before merge that used to reach staging or production, and what it would have cost if shipped.

After a fix, the agent's re-check verdicts never reached the customer's dashboard

Caught by: a real run replayed against the real backend, on the test's first run

Impact: customers never told which bugs their fix closed; hours of re-checking by hand

Every analysis kept its cloud machine alive after finishing, for 22 days unnoticed

Caught by: a replay that ran 120 s and was killed; 9.5 s once fixed

Impact: minutes of compute billed for nothing on every run; results delivered late

The follow-up run after a fix could not start: a date missing from the first run's record

Caught by: a replay that fails on main and passes on the branch

Impact: the fix-and-recheck loop broken for every customer

The agent asked for the same sandbox refresh 87 times in a row

Caught by: a dry replay of a recorded run, no call to a model

Impact: model calls and dollars burnt on every follow-up run; slower results

Progress reports rejected when a message was cut mid-character or over 100 kB

Caught by: an integration test on a real Postgres, before the commit

Impact: a run's progress and findings lost on the way to the customer's dashboard

A production safety check on run settings was skipped in one mode

Caught by: a unit test, in the session that wrote the change

Impact: a replayed run could pass for a real one: wrong findings shown to a customer

Public bugfix benchmarks indicate that allowing the agent to run the product dramatically improves performance, together with naming the bug to fix.

Our evaluation on more complex harnesses points in the same direction.

Benchmarks indicate running and naming bugs dramatically improves performance.

We reproduce it on complex products with the sandbox.

From the source alone a frontier model fixes under a tenth of real bugs. With the product running, several times more. With the bug described, nearly all. A verification harness is what moves an agent up that ladder. Our approach automatically builds one, regardless of system complexity.

Most repositories don't have a verification harness. The idea behind Heal's agent came from there: if an agent builds the missing piece when asked, one at a time, an agent can be asked to build all of them on a repository it has never seen, and then use them.

THE USER'S CODEBASETHE HEAL AGENT · ONE RUNWHAT THE RUN LEAVES BEHINDProduct repositoriesread, never written tothe source, on the developer'sbranchits own tests and typecheckan env example, maybea compose file, maybea CI workflow, maybeThe Heal agentbuilds, then uses, the harness1 map: reads the code, names the features and the vendors2 build: writes the sandbox (compose, stubs, seed, facts)3 verify: boots the product; a referee re-runs every check4 extend: adds the channel a feature needs: mail, a vendor5 refresh: on each commit6 hunt: finds defects on the running product, files each behind a failing testreadswritesfilesThe verification harnesssandbox/ and .heal/, beside the code1 tools: sandbox/a compose file with the storesa stub per vendorseed data and accountssign-in in test modePlaywright, page helpers, a mailcatcher client2 context: FACTS.md and .heal/how to boot, the health URLevery env key with a value that worksthe map: features, journeys, architecturethe run recordsthe defects found so far3 rules: what the referee re-runsboot and health probesthe failing test behind each defectthe product's tests

Figure 5. A run of the Heal agent produces a verification harness.

Heal's agent has built a verification harness on hundreds of products and filed tens of thousands of high-impact defects. We analyzed what the agent had to build to create the verification harness on a small subset (21 products):

Verification harnesses can be built manually, over time. Heal's agent does it automatically in order to run its bug and vulnerability hunter.

On a new repository Heal scores what is already there against its requirements, and builds the parts that are missing. A big part of that is the Heal application sandbox, the part of the harness that allows running the app in a way that closely mimics production. That application sandbox is a verification harness designed for identifying bugs and vulnerabilities, but can also be used by coding agents.

The end game is not one sandbox shape for every repository. It is a harness that fits each codebase: the tests it has, the ports its code already exposes, the vendors it calls, the way it boots. That harness belongs in the local workflow first, where the agent that wrote the change can run it before the push, and the same harness runs in Heal's cloud when a team wants the hunt and the referee on every commit. Try it on your repo, or read how it compares with code review bots in our benchmark.