Memory Agent

AI Agent

Ai Memory Management

Generative Ai Tools

Memory Is a Job, Not a Layer

Dr. Majid Fekri

23 min read·

Just now

--

Six ways the industry wires memory into AI agents, the evidence that each one breaks when the facts change, and what it takes to build memory that stays true.

Here is a result every team shipping agents should sit with.

In August, researchers ran 8,000 episodes of a tool-selection task where the right answer flips halfway through. Agents with no memory at all succeeded 31% of the time after the flip. Agents with append-only memory succeeded 21% of the time. So did agents with last-write-wins memory. (TEPA, arXiv:2608.07429)

Read that again. Once the world changed, the memory made the agents worse than having none.

That isn’t a bug in one product. It’s what happens when memory is built as a layer: a place where text goes in, gets embedded, and comes back out ranked by similarity. A layer has no clock and no judgment. It can’t notice that Monday’s fact died on Thursday, and nobody is responsible for noticing on its behalf.

In my last article I argued that agents need a memory agent, not a memory API. The response was generous, and the sharpest readers all asked the same thing: isn’t that just branding?

This is my answer. I’ll take the six most common ways teams give agents memory today, name who each one puts in charge of the truth, and show the evidence for where each breaks. Then I’ll follow the argument to its logical conclusion, with code, and hand you a test you can run on any memory system, including ours.

Disclosure: I’m CTO of the team behind Memanto, an open-source (MIT) memory agent. I name competitors below, but only through their own documentation and independent research, and I hold Memanto to the same test at the end.

Forget embeddings, graphs and context windows for a minute. Ask one question of any memory setup: who decides what is true now?

Long-lived memory is not a pile of text. It’s a stream of small, ongoing decisions:

Every memory system makes those decisions, or quietly leaves them unmade. The difference between systems is who makes them, when, and whether anyone can see or undo the result.

When you sort the market by that question, six setups fall out. Each puts a different party in charge of the truth. None of them is a party whose job is the truth.

Fig. 2. Six setups, six owners, and the evidence for where each breaks.

Setup 1: The vector store. Owner: nobody.

How it works. Every turn, document or tool result gets embedded. At query time, the top-k most similar chunks go into the prompt. It’s the default “long-term memory” in most framework tutorials, and it’s where most teams start.

Where it breaks. Similarity is not currency. “The API allows 100 requests a minute” and “The API now allows 500 requests a minute” are nearly the same vector. Ask about the rate limit and both come back. Whichever ranks higher wins, and usually both land in the prompt.

The TEPA authors put it precisely: stale evidence can remain highly relevant to the current query, giving similarity, recency and larger context only a weak signal that a fact has been superseded.

The evidence. In MemoryAgentBench, a UC San Diego benchmark built to test whether memory can revise itself when new evidence contradicts old, embedding-based retrieval agents scored 18–28% on single-hop updates and 4–7% when the update had to propagate through a second fact (Hu et al., arXiv:2507.05257). And Chroma’s context-rot study of 18 models found that even a single topically related distractor lowers accuracy (Chroma, 2025). A stale fact is the perfect distractor: same topic, wrong answer.

A vector store has no opinion about time. It has an opinion about cosine.

How it works. On every write, an LLM extracts facts from the message, looks up similar memories, and emits an operation: add, update, delete or nothing. Mem0 made this pattern popular, and many homegrown systems copy it.

Where it breaks. The decision is made at the worst possible moment: with one message of context, inside a model call, and destructively. When the model misreads scope, it doesn’t warn you. It deletes.

In August, a developer published a reproducible case: two compatible preferences (“async feedback for design reviews” and “sync calls for incidents”) were collapsed into a contradiction, and the design-review preference was silently deleted (DEV Community). Nothing threw. The agent just started routing feedback wrong, weeks later.

The evidence. The strongest evidence comes from Mem0 itself. Its v3 platform replaced the two-pass extract-and-merge pipeline with a single ADD-only pass in which, in its own words, nothing is overwritten or deleted, and it reports gains of 20+ points on its benchmarks from the change (Mem0 migration guide). Credit to them for changing course. When the best-known proponent of a design walks away from it, take the hint.

How it works. Keep everything. Old and new facts both persist, and retrieval ranking (similarity, recency, temporal cues) is supposed to surface the current one. This is where Mem0 v3 landed. It’s lossless, which is a real improvement over silent deletes.

Where it breaks. Append-only doesn’t resolve the contradiction. It moves it from the writer to the reader, and the reader is the worst possible owner: it meets the conflict mid-task, inside a token budget, with no authority to fix anything. Tomorrow it meets the same conflict again.

The evidence. This is the TEPA result from the top. In a controlled benchmark of 8,000 episodes, scored by a deterministic executor so it isolates the memory policy from any model’s quirks, success after a full reversal was:

On a preference-reversal stream it was worse: append-only scored 0.138, against 0.837 for no memory at all (Zhou et al., arXiv:2608.07429).

What won was revocation, which the paper describes as removing the stale precedent from ordinary retrieval while keeping it available for audit. Remember that sentence. It is most of this article’s argument.

Fig. 3. When the facts reverse, memory that can’t revoke is worse than no memory. Data: TEPA, Table 2.

Setup 4: The temporal knowledge graph. Owner: the pipeline.

How it works. Messages become episodes. An LLM pipeline extracts entities and relationships into a graph, and every fact carries a validity window. When new information contradicts an old fact, the system invalidates the old one instead of deleting it. Zep and its open-source engine Graphiti are the reference design (Zep docs).

This is the most serious temporal model in the field, and invalidate, don’t delete is the right instinct. I want to be clear about that.

Where it breaks. Two places. First, the invalidation is still an automatic model judgment made inside an extraction pipeline. Nobody is asked, and a wrong extraction quietly becomes a wrong graph. Second, there is a gap between writing and knowing. Zep’s own docs say added data is processed asynchronously and can take a few seconds to a few minutes, that a 100-episode import often takes minutes, and that a processed flag doesn’t guarantee every episode succeeded (Zep docs). In a fleet, that’s the window in which agent B acts without what agent A just learned.

The evidence. In MemoryAgentBench’s selective-forgetting test, Zep scored 7% on single-hop updates and 3% on multi-hop. That run used GPT-4o-mini as the backbone and Zep has shipped a lot since, which is exactly why these tests should be rerun in public, by everyone.

How it works. Memory lives in blocks pinned to the agent’s own context window, and the agent edits them through tool calls. This is the MemGPT idea, now Letta: a stateful agent is a system prompt, memory blocks, messages and tools, and the agent can modify its own memories (Letta docs).

Where it breaks. The worker is doing two jobs. Every memory edit spends the worker’s context and attention, which is the scarce resource context rot tells us to protect. Housekeeping only happens when the worker thinks of it, in the middle of something else. MemGPT scored 28% single-hop and 3% multi-hop on the same selective-forgetting test.

The evidence. The most telling evidence is what Letta built next: sleep-time agents, background agents that manage the primary agent’s memory blocks, triggered every five steps by default and still marked experimental (Letta docs). That is a memory agent in all but name. The team that invented self-editing memory concluded the worker shouldn’t do it alone. I take that as the strongest validation of this article’s argument.

How it works. CLAUDE.md, AGENTS.md, Cursor rules, system prompts: hand-curated standing instructions loaded into every session.

This is honestly the most trustworthy memory most teams have. It’s explicit, reviewable and in git.

Where it breaks. It’s a human doing the memory agent’s job by hand. There’s one file per tool, so Claude Code and Cursor drift apart. Nothing in it carries confidence, provenance or an expiry date. It grows until it rots, and because it’s loaded in full every session, it rots inside every context window. And it never learns from anything your agents observe.

Count the owners: nobody, the writer, the reader, the pipeline, the worker, and you. Six setups, and not one puts the truth in the hands of something whose job is the truth.

The benchmark agrees. Across every method MemoryAgentBench tested (long-context models, BM25 and embedding retrieval, structured RAG, Mem0, Cognee, Zep, MemGPT and MIRIX), the best multi-hop selective-forgetting score was 7%. The authors’ conclusion is one line: “Selective Forgetting still poses a significant challenge to all memory mechanisms.”

LongMemEval, one of the benchmarks memory vendors quote most, lists knowledge updates and temporal reasoning among its five core abilities, and found commercial assistants and long-context models losing about 30% accuracy on long interactions (Wu et al., ICLR 2025). Retrieval is not the hard part. Revision is.

Fig. 4. Every method tested, including four commercial memory systems, scored 7% or less on multi-hop updates. Data: MemoryAgentBench, Table 3.

The logical conclusion

Lay the argument out plainly.

Now look at what that owner does. It perceives (every write, every trace). It decides (type, confidence, conflict). It acts (retires, briefs, syncs). It answers to you. That is an agent by any definition you like. A layer can’t do any of it, because a layer has no goals and no clock.

That’s why “memory agent” isn’t branding. It’s the only shape the job fits.

If you actually believe that, some uncomfortable things follow.

The rest of this article is how we built Memanto to meet those obligations, written as a contract you can hold any memory system to.

Fig. 5. Worker agents write and read. The memory agent and you do the rest.

The Memory Agent Contract

Eight clauses. Each one comes with the evidence behind it, how Memanto meets it, and code you can run. Commands assume a local memanto serve on port 8000 and an active agent.

Why. Setup 2. A memory system that deletes on a model’s judgment will eventually delete something true, and you won’t hear about it.

How Memanto does it. A memory is either active or expired. There is no third state. Expiring a memory keeps its content and timestamps and writes three fields together: status, expired_at, and expired_by, which records why (a named policy rule, a retention entry, manual, or conflict-resolution). Restoring clears all three. Permanent deletion is a separate, explicit step that is off by default and never part of the nightly job.

Retire a memory; the reason becomes its audit stamp

curl -X POST "http://localhost:8000/api/v2/agents/support-bot/memories/$MEMORY_ID/expire"
-H "X-Session-Token: $TOKEN" -H "Content-Type: application/json"
-d '{"reason": "superseded-by-rewrite"}'

Changed your mind? Bring it back; the stamp is cleared

memanto memory restore $MEMORY_ID

Separation of d