Vektor 2.0.2: Memory benchmark LongMemEval results

Generative Ai Tools

Longmemeval

Ai Memory

VEKTOR 2.0.2: Memory benchmark LongMemEval results

Vektor Memory

8 min read·

Just now

--

Desk Trip Planner

VEKTOR 2.0.2 How we got there, via our custom Minimax matrix and a lot of testing

We measured where answers were being lost, changed one thing at a time, and kept a no-change control beside every test.

With the same open models throughout, the shipped build scored 89.2% on a paired check of 120 random LongMemEval questions, against 84.2% for the build before it. On the full 500 questions it scored 84.0%, the full run with the 30 preference questions taken from the shipped build (27 of 30).

Minmax: Each row is a candidate setting and each column is a kind of question:

Temporal, counting, updated facts, single-session lookups, preferences.

We run a candidate on a small set of questions from every column and score it by its worst column, since a setting that lifts one category by breaking another is not an improvement.

The winner is the row whose weakest column is highest. Each candidate runs next to a no-change control, because the same run twice moves a few answers by chance.

Each also runs against a guard sample of questions that were already right. Extraction is cached per question, so a test that only changes recall or the answering step takes minutes, not hours.

We tune on one set of questions and confirm on another. That is how the recall sizes (12, 80 and 120), the saving instruction, and the answering effort were chosen, and how the reranker, the paper-style expansion and duplicate removal were ruled out.

The newly built switch is in the preferences panel. The rest of the release upgrades are listed at the bottom.

LongMemEval is a public benchmark of long chat histories with questions about them. It has six categories and 500 questions. We ran every question through the real memory code, with a fresh database per question, and graded the answers with the benchmark’s own rules.

Every test had a control that changed nothing, because the same run twice moves questions by chance: on 90 missed questions, 8 turned right on a plain repeat. And every change was checked against a sample of questions that were already right, so a gain in one place could not hide a loss in another.

Larger clear benchmark image: https://vektormemory.com/research

Benchmark results

Dates written by code

The first deduction came from reading the misses. The model was asked to turn “last Friday” into a date, and it guessed. So code now does it, from the conversation’s own date: “last Friday” becomes 2023–03–10, “four weeks ago” becomes a target day, and the events near it are marked in a sorted timeline. Temporal questions went from 63.8% to 80.3% in the first lab run (graded by Llama 3.3 70B).

In Preferences the switch is called “Remember when things happened”. Every saved fact also starts with the day it was said, at no extra cost.

The second deduction came from counting the misses by cause. Of 61 multi-session misses, 52 were “how many” questions that had been shown only 12 memories. The answer was in the history, and only part of it was returned.

Recall size now depends on the question. Plain questions return 12 memories (it was 5). Counting and totalling questions about your own history return up to 80. Order, duration and relative-day questions return up to 120. A short tested check recognises each kind, so a plain lookup such as “how much does the plan cost” keeps the small size.

The results, on the same saved memory and the same judges: multi-session questions went from 54.1% correct at 12 memories to 75.9% at 80. Knowledge updates, where a fact changed over time, went from 74.4% at 5 memories to 85.9% at 12 (that curve graded by Llama 3.3 70B). Recall took the same time at 5 and at 12.

The third deduction: about 34 of the 90 questions still wrong were never saved at all. The saving step wrote summaries and dropped the named items. One added instruction now keeps every named item from both sides of a conversation, with list items and their position, and never keeps secrets or health records.

On the 52 previously missed questions it fixed 16, against 5 for a repeat with no change, and 39 of 40 right answers stayed right. It stores about 1.7 times as many memories and makes no extra model calls.

Our test harness had been calling the raw memory core, not the recall the app uses, which fuses keyword, meaning, graph and sentence search.

Measured through the app’s own recall, 29 of the 90 misses moved to right, against 8 for the control. Together with the saving change, 47 of the 90 were fixed against 8, and 30 of 32 guard questions stayed right.

On a paired check of 120 random questions, the shipped build scored 89.2% against 84.2% for the earlier build with the gpt-oss judge, and 85.8% against 82.5% on the majority of three judges.

The remaining misses are mostly lost counts. We traced them to the exact conversation turns that hold the evidence. Across 40 multi-session questions and 101 evidence turns, the saving step kept 94.1%, the app’s recall returned 77.2% at its normal size and 90.1% at double the size.

Then we tested the ideas that should have closed that gap. A cross-encoder reranker kept the same score at 24 memories and lost at 12. Expansion borrowed from the SmartSearch paper found no new evidence on top of our graph and sentence search.

Removing near-duplicates lifted the evidence returned from 77.2% to 84.2%, but the answers did not improve. A period computed in code for “in the past two weeks” made no difference. A preference-focused saving instruction scored no better than the control.

Each result narrowed the search. With nearly all the evidence in front of it, the answering model still miscounts, so counting is limited by the answerer, not by retrieval.

Preference questions are graded against a rubric, and two judges disagreed. We read all 30 answers ourselves. The gpt-oss judge agreed with a careful reading on 25 of 30, and the Qwen judge on about two in three. We graded preference with the judge that matched the audit and published the audit file. Under that judge, preference is 83 to 90%, close to the 90% other memory systems report.

We also stopped re-running everything. Weighting a small set of missed questions and a sample of right ones to their real sizes reproduces a full run (81.9% estimated against 82.0% measured), so a change can be judged in minutes.

Update to 2.0.2, open Preferences and turn on “Remember when things happened”. Then ask your memory a question about time, or how many of something you did.

Version 2.0.2 is live in the downloads page: https:www.vekmem.com

Run your own race, and keep your data sovereign