
Should agent-memory benchmarks test stale information?
An open evaluation platform for benchmarking Textual, Coding, and Multimodal Memory systems within a standardized, reproducible framework.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
An open evaluation platform for benchmarking Textual, Coding, and Multimodal Memory systems within a standardized, reproducible framework.
Each track keeps its own result table and detailed metric breakdown.
Long-context, persona, script, and conversation-memory benchmarks.
Memory retrieval and generation over image-rich or multimodal tasks.
Agent memory support for coding tasks and repository-context recall.
Open-source methods and commercial products both use participant-hosted Add/Search APIs. AML does not deploy repository-only submissions.
Operate publicly reachable Add/Search endpoints and submit the fixed API version for review.
Use the issued key to verify the synchronous Add/Search flow.
After smoke passes, submit the full scored evaluation.
Use the product pages to inspect rankings, run evaluations, and prepare an integration.
Public ranking with filters, dataset columns, and score bars.
Create eval jobs, watch progress, and inspect private results.
Eligibility, submission routes, required materials, timelines, rewards, and publication rules.
User guide, evaluation workflow, API contract, security, and result publication.
Add/search API contract, request fields, polling, and response schemas.
Follow releases, challenge updates, and participation announcements through our official channels.
Ideas, research, and updates from the team building Agent Memory Leaderboard.
Official perspectives on agent memory, evaluation, and the work behind the leaderboard.
Public rankings are separated by track. Use the selector inside the leaderboard frame to switch tables without mixing metric dimensions.
Open-source submissions are open. Submit complete materials by October 31, 2026 at 23:59 (UTC+8); evaluation closes November 4 at 23:59.
Public rankings are separated by track. Use the selector inside the leaderboard frame to switch tables without mixing metric dimensions.
Open-source submissions are open. Submit complete materials by October 31, 2026 at 23:59 (UTC+8); evaluation closes November 4 at 23:59.
Public rankings are separated by track. Use the selector inside the leaderboard frame to switch tables without mixing metric dimensions.
Open-source submissions are open. Submit complete materials by October 31, 2026 at 23:59 (UTC+8); evaluation closes November 4 at 23:59.
Create API-gated evaluation jobs, monitor task progress, inspect private results, and submit eligible Full runs for administrator review.
The second Full run unlocks 30 days after the first Full run is completed.
Choose a bound version. Use Run label to distinguish repeated evaluations of the same version.
Confirm each item before starting a public-board candidate run. The button remains locked until every item is checked.
Review completed Full runs from non-admin leaderboard keys. Approved results enter the public board; rejected results remain private.
Private scores stay scoped to the current leaderboard key. Admin runs remain separate from external review candidates.
Agent Memory Challenge 是 Agent Memory Leaderboard 的第二期公开评测活动,面向全球研究者、开源项目维护者和商业产品团队开放。参赛系统负责 Add 与 Search,平台统一完成 Answer、Eval、结果复核与公榜。
参赛免费,不设组队要求。参赛方承担自身 API、数据库、带宽和计算成本,平台承担统一 Answer、Eval 与评测编排成本。
评测类型决定系统接受什么任务;参赛组别决定结果展示在哪个榜单。两者是互相独立的两个维度。
第二期所有参赛组别都必须由参赛方自行部署 Add / Search API。Eval Key 是在线接口申请审核通过后获得的评测凭证。
提交固定版本、Add / Search 地址、鉴权方式和运行说明;公开仓库作为开放性与署名材料,不能替代在线接口。
提交固定产品版本、Add / Search 地址、鉴权和容量说明。无需公开内部实现,审核通过后获得 Eval Key,结果进入商业产品榜。
开源方法榜和商业产品榜都只通过参赛方自托管的 Add / Search API 接入。第二期不接受只提交 GitHub 仓库、Docker 镜像或启动说明并由 AML 代为部署。开源材料仍用于开放性、署名和复现审核。
请在提交申请前固定参评版本。正式 Full 评测受理后,不得因结果不理想更换版本或撤回。
系统名称与版本、联系人、机构或团队、拟参评类型、方法或产品说明、允许公开展示的信息,以及完整的提交说明。
已部署的 Add / Search API、鉴权与容量声明,以及公开仓库、固定 commit、原始工作引用和方法改动说明。
固定产品版本、Add / Search API、鉴权方式、评测专用密钥、容量限制、超时与限流说明。
选择评测类型和参赛组别,提交已部署的 Add / Search API、系统版本与完整材料。
平台审核在线接口和提交材料,通过后签发 Eval Key;AML 不代为部署参赛系统。
先验证 Add / Search、鉴权和端到端链路,再运行第二期正式 Full 评测。
请在 2026 年 10 月 31 日 23:59(UTC+8)前提交评测申请和完整材料。
本期评测于 2026 年 11 月 4 日 23:59(UTC+8)停止。Full 通常需要约 0.5—2 天,请提前提交稳定 API 并完成测试。
两个 Key 的签发方和用途不同,请勿将它们放入公开仓库、URL、截图、邮件正文或群聊。
第二届 Agent Memory Challenge 面向文本、代码与多模态三个赛道设置合计人民币 150,000 元奖金池。奖金仅面向符合赛事规则并通过审核的开源方法榜参赛团队,商业产品榜不参与奖金评选。
二等奖 2 名,每名人民币 8,000 元;三等奖 3 名,每名人民币 3,000 元。
奖金人民币 5,000 元。每个赛道奖金合计 50,000 元,三个赛道合计 150,000 元。
获奖资格、结果审核、奖金发放及其他具体安排,以主办方最终公告和赛事官网最新规则为准。
如出现平台异常或结果争议,请提交 Run / Job ID 与脱敏说明,进入官方复核流程。
提交申请前请先阅读 API 接入指南并准备完整材料。报名与评测问题可发送至 [email protected]。
Agent Memory Challenge is the second public evaluation cycle of Agent Memory Leaderboard. It is open to researchers, open-source maintainers, and commercial product teams worldwide. Participants provide Add and Search; the platform runs Answer, Eval, result review, and leaderboard publication.
Participation is free, with no team-size requirement. Participants cover the cost of their own APIs, databases, bandwidth, and compute; the platform covers unified Answer, Eval, and evaluation orchestration.
The evaluation type determines the tasks your system receives; the participant division determines where the result is listed. These are two independent dimensions.
In Cycle 2, every participant division must host stable Add / Search APIs. An Eval Key is issued after the submitted online endpoints and materials pass review.
Submit fixed Add / Search endpoints, authentication and operational details. A public repository supports openness and attribution review but cannot replace deployed endpoints.
Submit a fixed product version, Add / Search endpoints, authentication, and capacity details. Internal implementation may remain closed; an Eval Key is issued after approval and results enter the Commercial Products board.
Open-source and commercial entries integrate through participant-hosted Add/Search APIs. Cycle 2 does not accept repository-only or Docker-only submissions for AML deployment. Open-source materials remain part of openness and attribution review.
Freeze a clear evaluation version before applying. Once a formal Full evaluation is accepted, the version may not be replaced or withdrawn because of an unfavorable result.
System name and version, contact details, organization or team, intended evaluation type, method or product description, information approved for public display, and complete submission notes.
Deployed Add/Search endpoints, authentication and capacity declarations, plus a public repository, fixed commit, attribution of prior work, and method changes.
Fixed product version, Add / Search APIs, authentication, a dedicated evaluation credential, capacity, timeout, and rate-limit details.
Choose an evaluation type, participant division, and submission route, then provide a fixed system version and complete materials.
The platform reviews the live API and submitted materials, then issues an Eval Key. AML does not deploy participant systems.
Validate Add / Search, authentication, and the end-to-end path before the formal Cycle 2 Full evaluation.
The platform reviews the version, results, and compliance status before publishing the entry to its corresponding public board.
Submit the evaluation request and all required materials by October 31, 2026 at 23:59 (UTC+8).
Cycle 2 evaluation stops on November 4, 2026 at 23:59 (UTC+8). A Full run usually takes about 0.5–2 days, so submit and test a stable API early.
These credentials are issued by different parties and serve different purposes. Never place either credential in a public repository, URL, screenshot, email body, or group chat.
The second Agent Memory Challenge provides a total prize pool of US$22,000 across the Textual, Coding, and Multimodal tracks. Prizes are available only to eligible Open-source Methods teams that pass review; Commercial Products entries are not eligible.
≈ USD 3,000. The Textual, Coding, and Multimodal tracks are judged separately.
Second Prize: 2 winners, ≈ USD 1,200 each. Third Prize: 3 winners, ≈ USD 450 each.
≈ USD 750. The total prize pool across all three tracks is US$22,000.
Participant identity, system version, and all required materials are complete, truthful, and verifiable.
Add / Search, authentication, and the end-to-end path follow the current integration contract.
The formal evaluation and required tasks complete successfully without missing or duplicated results.
The code, image, or API used for the formal evaluation matches the declared version.
The version, results, and compliance status pass organizer review before the submission is considered valid.
Prizes are available only to eligible Open-source Methods teams that comply with the rules and pass review. Commercial Products entries are not eligible.
Textual, Coding, and Multimodal entries are ranked and awarded separately; scores are not added into a combined cross-track ranking.
Eligibility, result review, payment, and other arrangements are subject to the organizer's final announcement and the latest official rules.
Search must not generate final answers or disguise answers as memory records.
Do not share or retrieve evaluation memories across user IDs, tasks, samples, or teams.
When reusing a paper, repository, or codebase, identify the original authors, technical report, and every method change.
Hard-coding, data leakage, prompt injection, live human answering, result manipulation, and leaderboard abuse are prohibited.
Use the error details to correct the API, authentication, or runtime configuration, then validate again.
For platform failures or result disputes, submit the Run / Job ID with a redacted explanation for official review.
Read the API Guide and prepare all required materials before applying. For participation and evaluation questions, contact [email protected].
了解如何接入 Add / Search、运行统一评测、查看结果,并将符合条件的结果发布到公开榜单。
Agent Memory Leaderboard 使用统一的端到端流程评估不同记忆系统。数据库、索引和内部架构不作限制。开源方法榜(学术榜)的 Embedding 模型须使用 text-embedding-v4,LLM 相关组件须使用 gpt-4o-mini;Reranker 不作限制。商业产品榜(工业榜)不受上述模型限制。Add / Search 接口须符合现行规范,评测过程和检索结果须能够复核。
你只负责 Add / Search 环节。回答模型、提示词、评分器、数据集组合、Top K 和汇总规则由我们统一固定,使榜单分数尽可能反映记忆系统本身的能力。
你可以在 Textual、Multimodal 或 Coding Track 中查看 Overall、分项指标、系统版本和发布日期。不同 Track 使用的任务和指标不同,分数不宜跨 Track 直接比较。
实现 Add / Search → 提交准入申请 → 获取 API Key → 运行公开 smoke → 提交 full 正式评测 → 查看私有结果 → 进入公榜审核。
先提交 Evaluation Access Request,完成参测系统、版本、在线 Add / Search 接口和鉴权方式审核。第二期所有榜单均要求参赛方自行部署接口,AML 不接受仅提交仓库后代为部署。
选择申请新 Key,或输入已有 Key 为其增加版本;填写联系人、系统版本、Add / Search 接口、鉴权、容量和运行限制。开源方法另附仓库与来源披露。
审核通过后,新申请会签发 Leaderboard API Key;新增版本申请会直接绑定到原 Key,无需管理员再次手工录入接口信息。
平台直接校验参赛方提交的在线接口,并按照现行同步规范执行 Add → Search → Answer → Evaluate Smoke 流程,确认接口可用后再进入正式评测。
选择 full 前,必须逐项勾选提交清单:smoke 已通过、API 契约正确、运行说明完整,以及原创性披露和诚信承诺均已完成。
在 Evaluation 页面验证 API Key,选择已绑定的版本,填写唯一的 Run Label 后提交 full 评测。正式任务使用统一的数据集套件和 Top K。
任务依次执行写入、检索、回答、评分和结果汇总。页面会显示当前阶段、完成进度和错误摘要;我们同时保留复核所需的过程记录。
评测结果首先仅对绑定的 API Key 可见。成功完成的 full 任务会进入公榜资格校验和审核队列;审核通过后生成公榜提交记录。
允许复现已有论文或封装已有仓库,但必须披露原始作者、技术报告以及方法改动;未披露的重复代码可能被视为抄袭。发现重复代码时,平台会优先保护本人已披露并先提交的评测结果。重复提交相似或低质量代码、提示词注入、数据或结果操纵、恶意刷榜等行为,均可能取消参赛资格。
首次接入或接口行为变更后,先使用独立的兼容性 smoke 检查接口规范;接口稳定且版本准备完成后,再提交唯一的正式评测模式 full。
参赛选手配置 Add 和 Search 的接口地址,请求和响应格式按照现行规范固定,不随 URL 路径变化。生产环境建议使用 HTTPS。URL 中不得包含用户名、密码等凭据,也不得指向私有、回环或链路本地地址。
Add / Search 支持 Token、Bearer 和 X-Api-Key;none 仅用于公开 smoke。Health 接口通过无需鉴权的 GET 请求调用,返回任意 2xx 状态码即表示服务正常。
普通文本在达到 20 条消息或 2,000 个词中的任一边界时确定性分段,词数按冻结 Adapter 计数;Streaming 按增量会话/任务单元写入;代码保留仓库/会话轨迹,多模态按公开数据源单元与媒体限制处理。单次请求最长 30 分钟。
{ "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0", "messages": [{ "role": "user", "timestamp": 1704067200000, "content": "memory text" }], "user_id": "eval:run_abc123:locomo:conv-0", "session_id": "eval:run_abc123:sample:0" }
request_id必填。本次写入请求的唯一标识;成功响应必须原样返回该值。
必填。消息按原顺序排列,每条消息包含 role 和非空 content。文本与代码使用字符串;多模态可使用字符串或有序 text/image_url 内容数组。timestamp 可选,单位为 Unix 毫秒。
现行规范不发送 metadata、app_id、agent_id 或 async_mode;同步语义由 Add 完成写入后返回 HTTP 200 保证。
HTTP 200 { "success": true, "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0", "user_id": "eval:run_abc123:l
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。