
A small model with a review loop matches a larger one at 58% of the cost
On 20 ProgramBench tasks, GPT-5.6 Luna in a simple build-and-check loop reached the score of GPT-5.6 Sol's single shot for a median 58% of what Sol's run cost, and at Sol's cost it scored higher on 15 of the 20. A singl…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
On 20 ProgramBench tasks, GPT-5.6 Luna in a simple build-and-check loop reached the score of GPT-5.6 Sol's single shot for a median 58% of what Sol's run cost, and at Sol's cost it scored higher on 15 of the 20. A single Luna session told to play both roles stopped after one pass and scored no better than a plain single shot, and the loop never produced a badly failed run, where every single-shot arm did.
Coding models are trained to finish fast and sound right, which makes them confident by default. Ask one to build a program and it comes back with something that compiles and a note saying it's done, from the same model that did the work. In the runs below, 100 Luna sessions that had been told to check their own work returned after 4 to 20 minutes with about half of the hidden tests passing. Every one of them stopped on its own, well short of the six hours it had.
A review loop splits the two jobs. A builder works until it believes the task is done; then a checker, in a fresh session that hasn't seen the builder's reasoning, tries to break the result, and if it finds problems the builder gets the list and goes again. The checker is the same model with the same task statement; it just starts without the builder's transcript.
ProgramBench is a good place to measure whether that helps. Each task gives an agent a compiled program (svgbob, xz, samtools, direnv, 16 more) and its documentation, offline, and asks for a reimplementation from observed behaviour alone. Hidden tests then score the rebuild: 262 to 1,425 of them per task here. It's a fair stand-in for everyday work, and on our 20 tasks no published run passes every test; the best passes between 38% and 99%.
We ran the loop as a zeroshot graph with two nodes, a builder and a checker; zeroshot is our open-source runner for graphs of coding agents, and this is about the simplest graph it can run. One builder turn is a build, one checker turn a check, and the pair is a round. The builder keeps its session across rounds; every check starts fresh with only the task statement and the workspace. A rejection sends the checker's list of discrepancies back as the builder's feedback, and the loop ends when the checker accepts, after 50 rounds, or at six hours.
task
build GPT-5.6 Luna keeps its session across rounds
workspace
check GPT-5.6 Luna fresh session every round
accepted done
rejected: the discrepancies, each with a minimal reproduction, become the builder's feedback repeat: at most 50 rounds, or six hours
We wrote two prompts and froze them before the first run; neither was tuned to any task. Everything else the model sees is ProgramBench's own task statement and the runner's standard instructions for a builder and a checker.
Complete the task described in the input. Keep working and verifying your own work until you believe the result fully satisfies the task. If review feedback is provided, address every point without breaking what already works.
Independently judge whether the current workspace satisfies the task. Derive your own checks from the task statement and whatever it makes available; do not rely on the builder's tests, notes, or claims. Run the checks. Accept only with concrete evidence that nothing is wrong. Otherwise reject and list the most important discrepancies, each with a minimal reproduction.
Luna and Sol are two models in the GPT-5.6 line, Luna the small one and Sol the large one; we ran both through OpenAI's Codex CLI at its xhigh reasoning-effort setting. Luna costs $0.20 per million input tokens and $1.20 per million output tokens, Sol $4 and $20. Sol ran each task as a single shot (one build, no checker) five times. Luna's loop ran five times per task.
Why this comparison. If you're paying for the work, the question isn't which model scores higher but what a given level of quality costs. Luna costs about a twentieth of Sol per token, and the loop spends tokens on checks and repairs that a single shot never makes, so a loop's bill depends on how many rounds it takes. We asked, per task, what the loop had to spend to reach the score Sol's single shot gets there. The loop's five runs are scored build by build; the cost to match is the smallest budget at which the mean of their last builds within that budget reaches Sol's mean score, and we report it as a share of what Sol's single shot cost on the same task.
What we expected. If the checker catches bugs the builder missed, the loop should reach Sol's score before it has spent what Sol spends, with a wide margin on tasks where a cheap first build already lands close. If the small model can't do a task, no budget gets there. We expected the loop to come in cheaper on most tasks, not all: where a cheap first build lands far from the mark, 50 rounds of a small model may never close the gap.
What we found. On 15 of the 20 tasks the loop reached Sol's mean score for less than Sol's single shot cost, and across the 20 tasks the median is 58% of Sol's spend.1 On 3 tasks it needed more than Sol spent (130% to 193%), and on 2 (datasurgeon, xz) it hadn't reached Sol's score by its last build. The figure below shows each task.
Fixing the cost instead of the score gives the same picture. With each loop scored at the last build that fits inside Sol's mean cost on the task, 11 to 50 rounds in, the loop scored higher on 15 of the 20 tasks.2 On 7 of them (ditaa, parqeye, dstask, direnv, calcurse, gittype, gdu) the 95% intervals of the two means don't overlap; on the 5 tasks Sol leads, they do.3 Averaged over the tasks, the loop passes 69.5% of the hidden tests to Sol's 65.6%. What the loop does pay is wall-clock time: a median 69 minutes to reach Sol's score and 2.3 hours to spend Sol's budget, against 11 minutes for Sol's single shot (medians of the per-task averages).
Why this comparison. The cheapest explanation of the loop's lead is that the checker's instructions are good, and that any session told to build and then check would do as well. If that were true, the separate checker session would add little beyond the prompt.
What we expected. Two outcomes seemed possible. Either a single session given both roles keeps going for about as long as the loop does and lands near the same score, or it judges its own work fine and stops early. We leaned toward the second: the builder prompt already tells the session to verify its own work before it stops, and a second instruction to do the same thing seemed unlikely to change when it stops.
What we ran. We gave Luna the builder's and the checker's text above, verbatim, plus the runner's standard checker instructions and four sentences of glue; we call the whole thing the role prompt:
Work in rounds. In each round, act first as the builder and then as the checker, following the instructions for each role below. If the checker rejects, its list of discrepancies is the review feedback for the builder in the next round. Repeat until the checker accepts, then return.
The model, the task statement, the six-hour limit and the five runs per task were the loop's; the only change is one session instead of two.
What we found. Luna didn't loop. All 100 sessions returned on their own after 4 to 20 minutes (median 9), at about the price of one build. They scored 50.5% on average, 2 points below Luna's plain single shot (the loop's first build). At that spend the loop itself has finished exactly one build in all but one of the runs in the comparison,4 so this is a comparison of two prompts for one build, and the plain one wins it 14 tasks to 6. Everything the loop gains over a single shot comes from the extra rounds: 69.5% at Sol's budget, 71.1% after all its rounds.5
Why this comparison. If you ship agent-built software, the worst run matters as much as the average one. A single shot is one attempt, and attempts vary: the model commits to a design in its first minutes and may misread a flag along the way. A checker should catch the worst of that.
What we expected. The loop's runs should land closer to each other than any single shot's, and Sol's single shots, from the stronger model, should vary less than Luna's.
What we found. The figure below pools the 20 tasks, each run plotted as its score minus its arm's mean on that task; an arm is one of the four setups (Sol's single shot, Luna's single shot, the role-prompt sessions, the loop at Sol's budget). Scored at Sol's budget, 86% of the loop's runs land within 5 points of their task's mean; so do 72% of Sol's single shots, 65% of Luna's single shots and 62% of the role-prompt sessions.6
The loop's lead over Sol is in the tails. Four of Sol's 100 runs ended more than 15 points from their task's mean; the loop had one such run in 100, on the high side.7 Leave those out and the two arms are equally steady: 85% of Sol's remaining runs land within 5 points, 87% of the loop's. What the loop never did was fail badly: no loop run fell more than 15 points below its task's mean, where Sol's single shot had three such runs, Luna's two and the role prompt's two.
The loop we measured is the simplest thing that can be called a review loop: two nodes, two generic prompts that never changed, and a checker whose only advantage over the builder is a fresh start. On these tasks that was enough for a small model to reach a large model's score at a median 58% of its cost, and to do it without the badly failed runs that every single-shot arm produced. The trade is wall-clock time: reaching that score took the loop about six times as long as Sol's single shot, which matters for some uses and not for others. A prompt asking one session to do both jobs didn't produce the same effect, because the session stopped when it believed it was done.
That is a core hypothesis zeroshot is built on: an independent check catches what a builder's own assessment misses. The two-node loop is its smallest instance. A zeroshot graph can carry more than that, such as several builders working in parallel, checkers that specialize (one runs the tests, one reads the diff, one compares against the spec), verifiers with their own tools, and prompts written for the job at hand. We expect graphs built for a task to bring the cost-to-match number well below what a generic two-node loop gets.
Code, experiment files, every per-task result and the figure scripts are on the benchmark/programbench branch of the zeroshot repository.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。