Same open model, two harnesses: 24/25 vs. 19/25 on 25 real merged pull requests

Back to Blog Coding Agent Benchmark: AI Coding Agents on 25 Real Pull Requests

Coding agent benchmark: 25 real 2026 pull requests in five languages, graded by project tests. Octomind with an open model solved 24/25 at half the cost.

We wanted one number we could actually trust: if you hand a coding agent the kind of task a maintainer faces on a normal Tuesday — a real bug, a real feature request, in a real codebase — how often does it deliver a fix the project's own test suite accepts?

None of the public AI coding benchmarks could give us that number, so we built octobench. This post is the story of how, what broke along the way, and what the scoreboard says.

Two problems kept biting us with the popular LLM coding benchmarks.

Contamination. The well-known benchmarks are old enough that frontier models have seen the fixes — sometimes literally, commit by commit — in training data. At one point we caught a model running git show on a commit hash it had no business knowing about. Whatever that measures, it isn't problem-solving.

Taste-grading. Many benchmark tasks are graded by tests that assert an implementer's arbitrary choices: an internal variable name, the exact wording of an error message. An agent can write a maintainer-grade fix and fail because it phrased an error differently than the original author. That measures mimicry, not engineering.

So the design goals for our coding agent benchmark were: real tasks, fresh tasks, fair grading.

We harvested every task from a real, recently-merged pull request in a respected open-source project: werkzeug, click, anyio, pydantic, twig, carbon, symfony, guzzle, commonmark, uuid, rayon, chrono, bytes, serde-json, fmt, yaml-cpp, catch2, spdlog, eslint, fastify, undici, pino, pino-pretty.

That list is deliberate on two axes. Languages: five of them — python, php, rust, c++, js — because an agent that's great at python and lost in a CMake build is not a general coding agent. Freshness: every fix was merged in 2026, mostly after current model training cutoffs — several within days of harvesting. One case, chrono's reversed date iterators, was merged the same morning we picked it. A benchmark you can re-harvest as cutoffs advance is a benchmark that can't go stale.

The scenario mix runs from one-line crash fixes to multi-file features, and each case reconstructs the moment before the fix existed:

Before any case entered the benchmark it had to prove itself fail-to-pass: held-out tests must fail on the pre-fix code and pass with the real merged fix applied. No proof, no case.

For 20 tasks we ran a dedicated agent (octomind's developer:reverse-spec) over each merged commit to reconstruct the request that plausibly produced it, then curated by hand with one rule — the derivability rule: everything the hidden tests assert must be derivable from the prompt. Bug fixes got a short informal prompt, because any correct fix passes. Features whose tests pin public API names or exact output formats got a spec-tight version, because a real requester would state a wire format.

For the other 5 — one per language — we used the actual GitHub issue text, verbatim, trimmed only of fix-leaking sections (one author had helpfully written "I have a PR ready that does X"). That tests something different: turning a user-shaped bug report, sometimes with a screenshot instead of expected output, into a maintainer-grade fix.

A few favorites, by the skill they isolate:

Before the final run, three reviewers read every held-out assertion against every prompt. The audit caught real problems — in our benchmark, not the agents. One case's tests asserted the exact prose of an error message the issue never quoted; two frontier agents produced perfect fixes (one even picked the same error-code number as the maintainer) and both failed on wording. We replaced the case. Another case's hidden tests contradicted its own issue text. A third had its held-out tests inside a source file, so restoring gold tests wholesale would silently revert a correct fix.

The meta-lesson: benchmark infrastructure fails in ways that look exactly like model failures. Every anomaly we chased — a judge scoring 0 on a passing run, a "solved" case with an empty diff — deserved a real root-cause, and about half were ours.

Four agents, each at its stock, out-of-the-box single-agent invocation. No tuning, no custom prompts.

And because aggregate numbers hide the interesting stuff, here is every case. ✅/❌ is the verdict from the project's own held-out tests; j = judge score 0–100; then cost and wall time (including environment setup and verification).

The table rewards reading. Carbon's period-end sync took opus 164 minutes and cost opencode $43; commonmark's tab arithmetic — the screenshot-driven root-cause case — was solved only by octomind, in 2 minutes for a dollar; and yaml-cpp's binary emit styles is the one case octomind dropped — opencode failed it the same way with the same model, which is what a genuine model blind spot looks like, not a harness artifact. Every case has full per-run artifacts — agent traces, validation output, judge verdicts — in the repo.

1. The agent harness matters as much as the model. octomind and opencode ran the same model on the same endpoint at the same prices — a pure harness A/B. octomind solved 24, opencode 19, at half the cost. The difference is context discipline (opencode pushed 30M input tokens through one task where octomind needed 8M) and supervision: the cases opencode dropped are exactly the deep-root-cause, multi-trap ones where an unsupervised agent declares victory too early.

2. An open model beat Opus while paying full price for every token. glm-5.2 ran via Ollama cloud, which has no prompt caching — every agent turn re-bought its full context at list price. Opus, meanwhile, billed ~97% of its context re-reads at 1/10 cache rates. glm's dollar figure is its worst case, and it still came out ahead on solves, cost, and time. On any cache-enabled endpoint the cost gap becomes a chasm.

3. Speed has a thoroughness tax. Codex is remarkable — 1–4 minutes per case, $0.53 median — but all four of its failures are cases that punish not checking one more caller, one more surface, one more trap.

4. No task was impossible. Every case was solved by at least one agent, and the failure classes were clean: the trust-boundary case caught opus, codex, and opencode; the deep root-cause parser case caught everyone except octomind; and the one case both glm harnesses failed identically is a genuine model blind spot, not a harness artifact.

The other share is supervision. During dry runs an agent once "finished" a task by announcing "Now let me create a plan and implement the fix" — and stopping, with a zero diff. In a non-interactive run a text-only turn ends the session, so octomind added a deterministic guard: if a turn ends with no action while the agent's self-reported status is still in progress, it gets sent back to work. In the final run that failure mode was gone — and the cases opencode dropped are precisely the ones where nothing questioned an early declaration of victory.

Everything is public and pinned: the full per-case table and reproduction guide at the exact commit of this run, per-case agent traces and judge verdicts in the run artifacts, and the case-harvesting pipeline — the reverse-spec agent, the derivability rules, the fail-to-pass verifier — documented so the set keeps growing as training cutoffs advance.

The result we care most about isn't the ranking. It's the A/B: same model, same endpoint, five more tasks solved at half the cost. Model quality is table stakes now — the harness is where the leverage is. That's the thesis octomind is built on, and now we have the number.

20+ years in software engineering, the last decade deep in AI and retrieval systems. Since 2025, building Octomind — a session-first, multi-provider agent runtime that keeps context across hours of work. Writes from the production trenches: agentic coding, RAG pipelines, multi-model routing, and what actually breaks when AI runs at scale.

Hermes vs OpenClaw, rebuilt from both repos in October 2026: default security posture, memory, install cost, project health, and when to pick each one.

Claude Duet, Anthropic's proactive AI assistant: a morning brief, a cloud computer, approval before it sends or pays. Compared with ChatGPT Dots and Meta Muse.

Open source AI agent picks for coding, personal assistants, frameworks and visual builders: license, stars, releases and advisories checked Oct 2, 2026.