返回资讯中心
外部精选
软件工程
#开发者实时流

Turn your coding-agent sessions into private evals

evals-from-sessions is an agent skill. It turns your own coding-agent sessions into

Hacker News 最新EvgeniyZh2026/10/18 · 2026/10/186 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

evals-from-sessions is an agent skill. It turns your own coding-agent sessions into verified Harbor eval tasks, so you can compare agents, models and settings on your own work by price and time.

Guide: Turn your agent sessions into evals. Mine found a 2× cheaper setup · Leaderboard: ibragim.dev/leaderboard

The skill follows the six steps of the guide:

The skill stops when a task is ready for human review. Comparing setups across many runs (harbor run -p tasks/ ... -k 3) is your next step, not part of the skill.

Checkpoint 1: approve the candidate list (candidates.md).

Checkpoint 2: approve the task spec (provenance.md) before the agent builds the task.

Final review: check each finished task and its report.md before it enters your set.

Unattended runs write AWAITING_REVIEW at a checkpoint and stop.

Docker and git

Python 3.11+

Harbor: uv tool install harbor (any recent version; command examples use 0.24 syntax)

gitleaks 8

Credentials for one agent and model, for harbor check and the smoke run

Install

skills CLI (Claude Code, Codex, Pi and other agents):

npx skills add ibragim-bad/evals-from-sessions

Claude Code plugin:

/plugin marketplace add ibragim-bad/evals-from-sessions /plugin install evals-from-sessions@evals-from-sessions

Codex plugin:

codex plugin marketplace add ibragim-bad/evals-from-sessions codex plugin add evals-from-sessions@evals-from-sessions

Pi package:

pi install git:github.com/ibragim-bad/evals-from-sessions

Update: claude plugin update evals-from-sessions@evals-from-sessions, codex plugin marketplace upgrade evals-from-sessions, or pi update --extensions.

Manual: copy skills/evals-from-sessions/ into your agent's skills directory, for example ~/.claude/skills/, ~/.codex/skills/, ~/.agents/skills/ or ~/.pi/agent/skills/.

Start your agent in an empty work directory and ask, for example:

Use evals-from-sessions to turn my Claude Code sessions from the last month into eval tasks. Work directory: ./evals. Stop at each checkpoint.

Layout

skills/evals-from-sessions/ SKILL.md # the workflow references/harbor-contract.md # Harbor layout, network, commands assets/task-template/ # starting files for one task assets/leak-probe.sh # tries to recover the solution from the image assets/please-hack.j2 # optional "please game the verifier" smoke prompt

Safety

Session transcripts can contain API keys and private code. The skill works on a copy, scans it with gitleaks, and never writes secrets. Private-mode tasks contain a snapshot of your repository: keep them private. Publish only shareable tasks.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →