
Turn your coding-agent sessions into private evals
evals-from-sessions is an agent skill. It turns your own coding-agent sessions into
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
evals-from-sessions is an agent skill. It turns your own coding-agent sessions into verified Harbor eval tasks, so you can compare agents, models and settings on your own work by price and time.
Guide: Turn your agent sessions into evals. Mine found a 2× cheaper setup · Leaderboard: ibragim.dev/leaderboard
The skill follows the six steps of the guide:
The skill stops when a task is ready for human review. Comparing setups across many runs (harbor run -p tasks/ ... -k 3) is your next step, not part of the skill.
Checkpoint 1: approve the candidate list (candidates.md).
Checkpoint 2: approve the task spec (provenance.md) before the agent builds the task.
Final review: check each finished task and its report.md before it enters your set.
Unattended runs write AWAITING_REVIEW at a checkpoint and stop.
Docker and git
Python 3.11+
Harbor: uv tool install harbor (any recent version; command examples use 0.24 syntax)
gitleaks 8
Credentials for one agent and model, for harbor check and the smoke run
Install
skills CLI (Claude Code, Codex, Pi and other agents):
npx skills add ibragim-bad/evals-from-sessions
Claude Code plugin:
/plugin marketplace add ibragim-bad/evals-from-sessions /plugin install evals-from-sessions@evals-from-sessions
Codex plugin:
codex plugin marketplace add ibragim-bad/evals-from-sessions codex plugin add evals-from-sessions@evals-from-sessions
Pi package:
pi install git:github.com/ibragim-bad/evals-from-sessions
Update: claude plugin update evals-from-sessions@evals-from-sessions, codex plugin marketplace upgrade evals-from-sessions, or pi update --extensions.
Manual: copy skills/evals-from-sessions/ into your agent's skills directory, for example ~/.claude/skills/, ~/.codex/skills/, ~/.agents/skills/ or ~/.pi/agent/skills/.
Start your agent in an empty work directory and ask, for example:
Use evals-from-sessions to turn my Claude Code sessions from the last month into eval tasks. Work directory: ./evals. Stop at each checkpoint.
Layout
skills/evals-from-sessions/ SKILL.md # the workflow references/harbor-contract.md # Harbor layout, network, commands assets/task-template/ # starting files for one task assets/leak-probe.sh # tries to recover the solution from the image assets/please-hack.j2 # optional "please game the verifier" smoke prompt
Safety
Session transcripts can contain API keys and private code. The skill works on a copy, scans it with gitleaks, and never writes secrets. Private-mode tasks contain a snapshot of your repository: keep them private. Publish only shareable tasks.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。