FrontierMemory

Benchmarking continual learning through memory.

Read the blog

Models

Which models are the best at memory generation?

Opus 5.5 [max] 73.3%

$2.80

2 GLM-5.3 [max] 73.3%

$3.58

3 Kimi K3 [max] 66.7%

$3.73

4 GPT-6.1 Sol [max] 60.0%

$0.77

5

Gemini 4 Argon [high] 60.0%

$2.60

6 Muse Wonder [xhigh] 46.7%

$4.09

7

Grok 4.7 [xhigh] 33.3%

$3.02

GPT-6.1 Sol

Opus 5.5

Muse Wonder

Gemini 4 Argon

GLM-5.3

Kimi K3

Grok 4.7

Task effort:

low

medium

high

xhigh

max

0%20%40%60%80%$0.20$0.50$1$2 most efficient ↗

GPT-6.1 Sol Opus 5.5 Muse Wonder Gemini 4 Argon GLM-5.3 Kimi K3 Grok 4.7

Average task cost (log scale)

Memory generation at highest effort (Letta Dream) · Both task models across all task efforts (Letta Code) · Cost-efficient hull per model shown

Which harnesses are the best at memory generation?

Letta Dream 73.3%

$2.80

2 Prime Agent 60.0%

$3.74

3 OpenAI Agents SDK 40.0%

$0.82

4 Claude Managed Agents 33.3%

$3.21

5

Hermes Agent 26.7%

$2.79

Letta Dream

OpenAI Agents SDK

Hermes Agent

Prime Agent

Claude Managed Agents

Task effort:

low

medium

high

xhigh

max

0%20%40%60%80%$0.20$0.50$1$2 most efficient ↗

Letta Dream OpenAI Agents SDK Hermes Agent Prime Agent Claude Managed Agents

Average task cost (log scale)

Memory generation at highest effort · Both task models across all task efforts · Cost-efficient hull per harness shown

Existing memory benchmarks measure recall of past facts. FrontierMemory is the first benchmark that measures the quality of memory. Agents are given access to trajectories of past experience and tasked to generate memory, which is then scored by how much it helps an agent solve downstream tasks.

Agents are given access to trajectories of agent experience and generate memory.

Memory is evaluated based on how much it helps solve downstream tasks.

We select challenging coding and knowledge-work tasks from public benchmarks, including HarborIndex, DeepSWE, and SkillsBench, both for source transcripts and for downstream tasks. Some tasks are modified so solving them requires knowledge available only in the source transcripts.

DeepSWEAwilix Release Tier (Implicit) Add async initialization to Awilix, where the version bump is determined by applying a policy learned from earlier tasks to a release tier found in project metadata.

Build a Word2Vec component whose interface, preprocessing, and output contract must be inferred from previous runs.

Compute DAPT2020 network statistics using lessons from failed analyses about counting, entropy, and threat detection.

Each memory-generation harness and model is ranked by the higher downstream task solve rate of Opus 5.5 and GPT-6 Sol, both at max reasoning. When only one task model is available, its score is used.

Existing memory benchmarks mostly measure recall of past facts. FrontierMemory is the first benchmark to measure memory by how much it helps an agent solve real tasks.

The agent sees the source transcripts. These cover the downstream tasks as well as other, unrelated tasks, and each transcript is labeled with whether the attempt succeeded or failed.

No. Web browsing and online solution lookup are not allowed during memory generation or task execution. Agents work from the provided trajectories and task environment; model API calls remain available.

FrontierMemory primarily compares models on how well they generate memory. By default, each model generates memory in the Letta Dream harness and is evaluated in Letta Code. Submissions can also bring their own memory generation and task harnesses, to maximize compatibility and allow harness comparisons.

To submit a harness, it must have a memory system it uses to create memories or skills between sessions.