返回资讯中心
外部精选
软件工程
#开发者实时流

harness-evals 发布:固定模型对比不同 agent 设计

来源标题:Harness-evals. compare agent designs with the model held fixed

开源 Python 库 harness-evals 用于在固定模型的前提下评估 agent harness 或工作流。用户将用例描述为带检查的任务,通过简短适配器接入各 harness,即可获得多次运行的区间指标,包括成功率、可靠性、成本、延迟和模型用量。它会在两个指标上绘制各设计,标记出没有其他设计能在两项上同时超越的方案,并判断应采纳、拒绝还是继续研究某项改动。

Hacker News 最新slavadubrov2026/11/16 · 昨天30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

A small Python library for evaluating an agent harness or a workflow with the model held fixed. You describe your use case as tasks with checks, connect each harness through a short adapter, and get repeated-run metrics with intervals: success, reliability, cost, latency and model usage. It plots the designs on two metrics at once, marks the ones no other design beats on both, and decides whether to adopt, reject or keep investigating a change.

It is the companion code for the article on evaluating agent harnesses in the Recursive Harness Engineering series. The agent-harness-lab installs it to evaluate five designs of one support assistant, from a plain agent to a workflow with no agent.

Two figures answer the questions a table cannot: which harness gives the best trade-off, and where each one fails.

uv add "harness-evals @ git+https://github.com/slavadubrov/harness-evals@v0.5.1"

or

pip install "git+https://github.com/slavadubrov/harness-evals@v0.5.1"

Quickstart

The example runs offline in a second. It is in the repository, not in the installed package:

git clone --depth 1 --branch v0.5.1 https://github.com/slavadubrov/harness-evals cd harness-evals uv run python examples/refund_desk.py out

It prints the table and the decision shown below, and saves the runs and both figures to out/. Its "agent" is a simulated model with seeded mistakes, so the numbers describe the simulation only. The same results from the saved runs:

uv run harness-evals summarize out/episodes.jsonl --usecase examples.refund_desk:USECASE uv run harness-evals compare out/episodes.jsonl --usecase examples.refund_desk:USECASE --base agent --candidate workflow

A model benchmark keeps the harness fixed and changes the model. A harness evaluation does the opposite: the model, its endpoint and settings, the tasks and the checks stay fixed, and only the harness changes. Then a difference in the results belongs to the harness.

This is a template: Environment, run_my_agent and my_workflow stand for your code. examples/refund_desk.py is the runnable version.

from harness_evals import Episode, Task, UseCase, at_most, did, never, run, says

USECASE = UseCase( name="refund-desk", tasks=[ Task( "refund-ok", "ana: order A1 arrived broken, please refund it", (did("refund", order="A1"), at_most(1, "refund"), says(r"refunded")), ), Task("refund-late", "ana: refund order A2", (never("refund"), says(r"30 days"))), ], gates={"violations": ("<=", 0), "success": (">=", 0.8)}, # the bar a harness must clear guardrails={"latency_p95": ("<=", 1.0)}, # a candidate may be at most 1 s slower )

def my_agent(tasks, trial): episodes = [] for task in tasks: env = Environment() # records what really happened reply = run_my_agent(task.input, env) # your harness episodes.append( Episode( task.id, reply=reply, actions=env.log, # a list of Action(name, args) the environment recorded cost_usd=env.cost, latency_s=env.seconds, model_calls=env.calls, ) ) return episodes

episodes = run( USECASE, {"agent": my_agent, "workflow": my_workflow}, trials=10, frozen={"model": "provider/model-id", "temperature": 0}, )

Print the results, and compare the two harnesses:

from harness_evals import by_harness, compare, comparison, summarize, table

by = by_harness(USECASE.grade(episodes)) # {"agent": [...], "workflow": [...]} summaries = {h: summarize(gs, USECASE.metrics) for h, gs in by.items()} print(table(summaries, extra=list(USECASE.metrics))) print(comparison(compare(USECASE, by["agent"], by["workflow"])))

summarize takes the grades of one harness. Pass USECASE.metrics, or your own metrics are left out.

did, never, at_most, says, never_says, ended, state and judged cover the common obligations. For anything else, wrap your own function in a Check:

Check("reply under 80 characters", lambda e: e.reply is not None and len(e.reply) < 80)

Return a real True or False: None means the evidence is missing and is reported as undecided, so write bool(re.search(...)), not re.search(...). A check that raises or returns any other type is undecided too. This also applies to saved state and judge verdicts: strings such as "false" cannot pass. never and at_most are critical by default and the others are not; pass critical=True or critical=False to change that.

Adapters receive independent copies of the tasks, and the runner snapshots each returned episode. Duplicate or unrequested episodes raise an error. Missing episodes and adapter exceptions become infrastructure failures; an exception marks every task in that batch as failed with unknown cost, keeps the exception's message in Episode.error and warns once per harness, so a bug in an adapter does not pass as a low score. trials must be a positive integer.

examples/refund_desk.py prints this, among other lines. Its cost, latency and usage are simulated:

Harness Success pass^k Violations $ per task $ per success p50 latency, s p95 latency, s Calls per task Tokens per task Infra errors Undecided
agent 0.90 (0.78–1.00) 0.60 (k=10) 2 $0.00022 (0.00020–0.00023) $0.00024 (0.00021–0.00028) 5.52 (4.90–6.09) 8.42 (7.57–8.69) 3.62 (3.38–3.84) 3475 (3245–3686) 0 0
workflow 0.80 (0.40–1.00) 0.80 (k=10) 0 $0.00006 (0.00006–0.00006) $0.00008 (0.00006–0.00015) 1.92 (1.84–2.04) 2.17 (2.12–2.17) 1.00 (1.00–1.00) 960 (960–960) 0 0

workflow against agent on 5 tasks: success -0.100 (-0.420 to +0.120). Better on 1 task, worse on 1 task. Candidate minus base: cost_per_task -0.00016 (-0.00017 to -0.00014); latency_p50 -3.60 (-4.17 to -2.96); latency_p95 -6.25 (-6.57 to -5.40); calls_per_task -2.62 (-2.84 to -2.38); tokens_per_task -2515 (-2726 to -2285). Decision: adopt. the base misses the bar: violations = 2, needs <= 0

Use cases

Every rate, cost, latency, usage and custom metric also comes with <name>_ci, a 95% bootstrap interval. It resamples whole tasks with all their runs, 2,000 times, and recomputes the metric on each resample: success and pass^k average over the drawn tasks, the other metrics are computed on their pooled runs. Repeated runs of one task are not independent evidence about other tasks, so the intervals never resample runs alone. Define custom metrics per run, as a share or a mean over runs: a metric that groups runs by task would see a task drawn twice as one. A metric that cannot be computed on a resample is undefined there, and if more than 2.5% of resamples are undefined, its interval is unknown (None).

Add the metrics that matter for your use case through UseCase.metrics, for example tool calls per task, how often a reply names an id, or a judge's score. Custom metrics cannot replace required metrics.

Choose the two metrics that matter most for the decision, such as success and cost, or success and latency, and plot every harness on them with an interval on each axis. A harness is on the frontier when no other harness is at least as good on both metrics and better on one; any harness off it has an alternative that is better on one metric and no worse on the other.

from pathlib import Path

from harness_evals import frontier_svg, pareto

pareto(summaries, "cost_per_task", "success") # ['agent', 'workflow'] Path("frontier.svg").write_text(frontier_svg(summaries, "cost_per_task", "success"))

The first figure at the top of this page is that plot for the example.

Directions are built in: higher is better for success, pass^k and custom metrics, lower for cost, latency, calls, tokens and error counts. Pass lower=["my metric"] for a custom metric where smaller is better. The frontier uses point estimates, so read the intervals before trusting the order of harnesses whose intervals overlap. When the harnesses are close variants of one design, pass zero=False: each axis then starts just below the data instead of at zero. A harness that lacks one of the two metrics is left out and named under the title. If either axis is a cost metric, any harness with unknown_cost > 0 is also omitted. Its measured-cost mean remains available in the summary.

A summary says how often a harness fails; a failure map says where. Each task is a row. With by="checks" (the default), each kind of check is a column, so did refund(order='A1') and did refund(order='B1') share the column did refund; a cell shows the share of the task's runs that failed that check, and a dot marks a check the task does not have. With by="runs", each run is a column. Every harness gets its own panel with the same rows, so you can see which harness fails which check on which task. Hovering over a cell gives the count when the SVG is opened on its own; an image in a README shows no tooltips.

from harness_evals import failure_map_svg

Path("failures.svg").write_text(failure_map_svg(USECASE, by)) # by = by_harness(...) as above

The second figure at the top of this page is that map for the example.

Pass group= to name the columns another way; it receives a check's name. Use by="runs" for the run grid. failures(grades) returns the same counts per check name as a dictionary.

compare(usecase, base, candidate) requires one harness per input, the same tasks and the same number of trials per task, without duplicate task/trial pairs. It pairs the harnesses task by task and gives every compared metric a paired 95% interval of candidate minus base: success, cost per task, both latency percentiles, calls, tokens, the use case's metrics and its guardrails. Each resample draws the same tasks for both harnesses. Equal task counts alone do not prove that either input covers the intended task split; the caller must supply complete data. The saved-run CLI checks this against the manifest.

The gates come first: a base that misses one is replaced by any candidate that clears them all. Between two harnesses that clear the gates, success and cost per task decide whether the candidate is better, and guardrails limit how much other metrics may get worse. A guardrail is written like a gate but on the difference, candidate minus base: guardrails={"latency_p95": ("<=", 2.0)} lets p95 latency rise by at most 2 seconds, and {"names id": (">=", -0.1)} lets a higher-is-better metric drop by at most 0.1. Put an absolute limit, such as a maximum latency, in the gates. The rule decides:

Steps 1 and 2 read point estimates: a gate ignores the interval. Steps 3 and 4 need at least six tasks; with fewer, the result is inconclusive. Six is a minimum policy, not a guarantee that the bootstrap interval has 95% coverage.

A cost gate fails while any run's cost is unknown, and a gate on a metric that could not be computed fails too. A cheaper-candidate decision requires complete costs for both harnesses, and a guardrail on a metric that could not be compared keeps the base. margin must be finite and nonnegative; zero allows no success loss. A zero-width success interval cannot justify cost-only adoption; widths up to 1e-15 count as zero to allow for floating-point roundoff. Identical observed task success rates produce [0, 0], but that result does not bound the loss on new tasks. The comparison keeps the interval as a descriptive result and returns inconclusive unless an earlier rule decides. This check does not validate bootstrap coverage for other cases. Comparison.deltas holds every compared metric's difference and interval, and decide(usecase, base_summary, candidate_summary, deltas) applies the rule to them.

Pass out="runs/evaluation" to run to save episodes.jsonl and manifest.json. With the use case in my_usecase.py as USECASE, in the folder you run the command from:

harness-evals summarize runs/evaluation/episodes.jsonl --usecase my_usecase:USECASE harness-evals compare runs/evaluation/episodes.jsonl --usecase my_usecase:USECASE --base agent --candidate workflow harness-evals frontier runs/evaluation/episodes.jsonl --usecase my_usecase:USECASE --x latency_p95 > frontier.svg harness-evals failures runs/ev

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →