返回资讯中心
外部精选
软件工程
#开发者实时流

Attest 发布 AI Agent 测试框架,可校验工具调用与外部状态变更

来源标题:Show HN: Attest – Test what AI agents do, not just what they say

Attest 1.1.0 是一个面向 AI Agent 的 Python 和 TypeScript 测试框架,支持用 pytest 或 Vitest 检查回复、工具调用序列及外部状态变化。框架推荐先做确定性检查,仅在无法表达时引入模型裁判,并提供 capture、严格 replay 和 gate 三步回归流程。检查结果分为通过、失败或未评估,失败默认阻断。

Hacker News 最新tommathews2026/11/15 · 2 天前21 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Get Started (TypeScript) Run the demo (refund-desk)

Attest 1.1.0 is a testing framework for AI agents in Python and TypeScript. Check a reply, a sequence of tool calls or a change to external state with pytest or Vitest. Start with deterministic checks; add a model judge only for criteria those checks cannot express.

Section titled “rec → replay → gate”

A support agent refunded a customer twice after a payment timeout. Its output-only tests stayed green. That synthetic incident — INC-1042 — becomes a regression test in three moves:

attest capture records the trace and model exchanges, applies redaction and drafts contracts from the run. A fixture marked consistent with the recording is not proof that the behavior is correct: review the fixture and choose independent business invariants.

Strict replay runs your agent’s real code against recorded model responses through a local proxy, with zero live model calls for matching requests. It removes live-model variability, not other sources of flaky tests; tool dependencies and test isolation still need control.

Checks report pass, fail or not_evaluated with a reason. Failed tests block by default (tests_failed: block); not_evaluated warns unless the policy sets not_evaluated: block. Require the policy check in branch protection. The gate covers the contracts and baselines you supply, not every possible regression.

The tour follows the double-refund ledger, reviewed contracts, a blocked revert PR, mutation scores and the HTML viewer. Its outputs illustrate the synthetic demo; the demo README has the current executable walkthrough.

expect(trace).tool_cardinality("issue_refund", per="order_id", at_most=1)

attestExpect(trace).toolCardinality("issue_refund", { per: "order_id", atMost: 1 });

One line of behavior contract states the rule INC-1042 broke — over the trace’s tool calls, not the reply text.

Section titled “Six evaluated layers plus plugins”

The replay and gate run on deterministic checks. The layers are the full ladder of checks a contract or test can use, from free and instant to LLM-judged. Plugins add your own assertions. The table is generated from attest-engine capabilities --json.

Section titled “What it looks like”

Python

Section titled “Python”

from attest import agent, expect

@agent("weather-agent")

def weather_agent(builder, query: str) -> dict:

Recorded sample data: no model or weather API is called.

builder.add_tool_call(

"get_weather",

args={"city": "Tokyo"},

result={"temperature_c": 22, "conditions": "Sunny"},

)

return {

"message": "Current weather conditions in Tokyo: sunny, 22°C.",

"structured": {"city": "Tokyo", "temperature_c": 22},

}

def test_weather_agent(attest):

result = weather_agent(query="What's the weather in Tokyo?")

chain = (

expect(result)

.required_tools(["get_weather"]) # L3, free

.output_contains("Tokyo") # L4, free

.output_matches_schema({ # L1, free

"type": "object",

"required": ["city", "temperature_c"],

"properties": {

"city": {"type": "string"},

"temperature_c": {"type": "number"},

},

})

)

attest.evaluate(chain)

dict: # Recorded sample data: no model or weather API is called. builder.add_tool_call( "get_weather", args={"city": "Tokyo"}, result={"temperature_c": 22, "conditions": "Sunny"}, ) return { "message": "Current weather conditions in Tokyo: sunny, 22°C.", "structured": {"city": "Tokyo", "temperature_c": 22}, }def test_weather_agent(attest): result = weather_agent(query="What's the weather in Tokyo?") chain = ( expect(result) .required_tools(["get_weather"]) # L3, free .output_contains("Tokyo") # L4, free .output_matches_schema({ # L1, free "type": "object", "required": ["city", "temperature_c"], "properties": { "city": {"type": "string"}, "temperature_c": {"type": "number"}, }, }) ) attest.evaluate(chain)">

Save this as test_weather_agent.py, install with uv add "attest-ai>=1.1.0" pytest, then run uv run pytest -v. Every assertion above is deterministic and requires no API key. For semantic checks, add output_similar_to or passes_judge after configuring the corresponding embedding or judge provider.

Section titled “TypeScript”

import { agent, attestExpect } from '@attest-ai/core';

import { evaluate } from '@attest-ai/vitest';

import { test } from 'vitest';

const weatherAgent = agent('weather-agent', (builder) => {

// Sample data: no model or weather API is called.

builder.addToolCall('get_weather', {

args: { city: 'Tokyo' },

result: { temperature_c: 22, conditions: 'Sunny' },

});

return { message: 'Current weather conditions in Tokyo: sunny, 22°C.' };

});

test('weather agent returns weather data', async () => {

const result = weatherAgent({ query: 'What is the weather in Tokyo?' });

const chain = attestExpect(result)

.requiredTools(['get_weather']) // Layer 3: Free

.outputContains('Tokyo') // Layer 4: Free

.outputNotContains('error'); // Layer 4: Free

await evaluate(chain);

});

{ // Sample data: no model or weather API is called. builder.addToolCall('get_weather', { args: { city: 'Tokyo' }, result: { temperature_c: 22, conditions: 'Sunny' }, }); return { message: 'Current weather conditions in Tokyo: sunny, 22°C.' };});test('weather agent returns weather data', async () => { const result = weatherAgent({ query: 'What is the weather in Tokyo?' }); const chain = attestExpect(result) .requiredTools(['get_weather']) // Layer 3: Free .outputContains('Tokyo') // Layer 4: Free .outputNotContains('error'); // Layer 4: Free await evaluate(chain);});">

Save as weather.test.ts and install with npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4". Run npx attest doctor once to download and check the engine before npx vitest run: a cold first download can exceed Vitest’s default timeout. This test makes no provider calls.

Section titled “Features”

Framework Adapters

Provider and framework adapters for Python and TypeScript, each with a maturity tier - the table below lists them.

Beta: trace delegation chains and check parent-child relationships, ordering and cross-agent data flow.

Beta: use mock tools, seeded faults and user personas. Your application’s own network calls still need mocks or replay.

Experimental: sample application traces and alert on hard failures or score drift through Slack or webhooks.

Stable workflows include capture, strict replay, behavior contracts, world effects and CI policies and baselines. The overview links every feature and its maturity tier.

From attest-engine capabilities --json at engine 1.1.1.

Section titled “How it compares”

Capability AttestDeepEvalPromptfooRagasLangWatchLangSmith SDK languages Python and TypeScriptPython (pytest) and TypeScript (Vitest)CLI and library; "Use Python, Javascript, or any other language"Not assessedPython and TypeScriptNot assessed Framework and provider integrations named on the page 10 Python and 7 TypeScript adapters11 for Python and 6 for TypeScript, including LangChain, OpenAI, Anthropic, LlamaIndex, CrewAI, Google ADKNot assessedLangChain, LlamaIndex, "and more"Not assessedOpenAI, Anthropic, CrewAI, Vercel AI SDK, Pydantic AI Runs inside the unit-test runner pytest and Vitestpytest and VitestNot assessedNot assessedNot assessedNot assessed LLM-as-judge Yes (layer 6)Yes (GEval)Not assessedNot assessedNot assessedNot assessed Deterministic assertions and per-layer cost classes Yes (layers 1-4 free; layers 5-6 depend on the provider)Not assessedNot assessedNot assessedNot assessedNot assessed Multi-agent traces and simulation Yes (beta)Not assessedNot assessedNot assessedNot assessedNot assessed "Not assessed" means the cited page does not say. Primary sources, accessed on the date shown:

Section titled “Get started”

Terminal window

Python

uv add "attest-ai>=1.1.0" pytest

TypeScript

npm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"

npx attest doctor

=1.1.0" pytestnpm install -D "@attest-ai/core@>=1.1.0" "@attest-ai/vitest@>=1.1.0" "vitest@^4"npx attest doctor">

Check the Python quickstart or TypeScript quickstart for a complete walkthrough.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →