Benchmarking retrieval for agents on messy real-world company knowledge

chunk_1 Pricing page "SSO is included in the Team and Enterprise plans."

chunk_2 SSO setup guide "To enable SSO, go to Settings > Security, choose your identity provider, and paste in its metadata URL."

chunk_3 Community forum post "You get SSO on the Team and Enterprise plans."

chunk_4 Changelog "Dark mode is now available in the dashboard."

chunk_1 Pricing page "SSO is included in the Team and Enterprise plans."

chunk_2 SSO setup guide "To enable SSO, go to Settings > Security, choose your identity provider, and paste in its metadata URL."

chunk_3 Community forum post "You get SSO on the Team and Enterprise plans."

chunk_4 Changelog "Dark mode is now available in the dashboard."

chunk_1 Pricing page "SSO is included in the Team and Enterprise plans."

chunk_2 SSO setup guide "To enable SSO, go to Settings > Security, choose your identity provider, and paste in its metadata URL."

chunk_3 Community forum post "You get SSO on the Team and Enterprise plans."

chunk_4 Changelog "Dark mode is now available in the dashboard."

chunk_1 Pricing page "SSO is included in the Team and Enterprise plans."

chunk_2 SSO setup guide "To enable SSO, go to Settings > Security, choose your identity provider, and paste in its metadata URL."

chunk_3 Community forum post "You get SSO on the Team and Enterprise plans."

chunk_4 Changelog "Dark mode is now available in the dashboard."

Score

Pass. The criterion requires chunk_1 and chunk_2, and both were returned.

Precision: 2 of 4, or 50%. chunk_3 and chunk_4 are not in the criterion.

The forum post is not in the criterion because of source preference. It answers the same part of the query as the pricing page, which plans include SSO, and the pricing page has more source authority. The changelog is simply not relevant. Both count against precision, but for different reasons: one is irrelevant, the other is relevant but not preferred.

In total, we created 1,000 eval cases for the Company Knowledge Bench, and they are what the results later in this post are based on.

Labelling a thousand eval cases by hand is not feasible. Sampling them properly makes it even harder: the queries come from every type of source we ingest, internal as well as public corpora, and the industries we serve. So to achieve this scale, the labelling has to be done by agents.

Of course, letting agents create the benchmark only works if they do it correctly. A wrong criterion means a wrong score for every retriever we test against it. So before the agents can build our benchmark, we need a second benchmark that measures how well they build it: a benchmark for benchmark creation.

That second benchmark is a set of 170 eval cases that humans labelled completely by hand, following the same benchmark labelling handbook. Each has a real query, a corpus, and a retrieval criterion a person wrote after working through the corpus themselves, so together they encode the handbook’s rules.

So the process was: build agents that can do the same task the humans did for this set, run them on its queries, and compare the criteria they write with the human ones. We kept improving the agents until they reached a high enough agreement rate with the human labellers. Only then did we let them generate the 1,000 eval cases of the benchmark from production queries.

The labelling is split between two kinds of agents:

Candidate agents scan the full corpus for chunks that might belong in the retrieval criterion. They are built for recall and return a large set of possibly relevant chunks.

Criteria agents take those candidates and turn them into the actual retrieval criterion, following the rules in the benchmark labelling handbook.

The agents do not have to reproduce the human criteria exactly for the benchmark to give a good signal, but they do have to come close. Getting there took many iterations, on the agents and on the handbook itself. We made sure the agents reached sufficient agreement with the human labellers, and validated them extensively, before moving on to generate the benchmark.

The result is labels that are close to what a human would write, and unlike hand labels they scale well beyond the 1,000 eval cases we generated for this benchmark.

We ran seven retrievers on the Company Knowledge Bench. All of them search the same corpora, ingested and chunked the same way, so the only thing that differs is the retrieval strategy.

These are the traditional RAG approaches of the last few years. They run the same steps for every query, with no model deciding what to do next.

Hybrid search. The baseline most RAG pipelines start from. The query is matched against a keyword index and an embedding index at once, using gemini-embedding-001 for the embeddings, and the top 15 chunks are returned.

Hybrid search with a reranker. The same search, but it retrieves the top 100 chunks and passes them through a reranker, Voyage AI’s rerank-2, which reads the query and each chunk together and scores how well the chunk answers it. The top 15 after reranking are returned.

Query decomposition, hybrid search, and a reranker. Before searching, a model, gpt-6.1-luna, splits the query into up to three sub-queries. Each sub-query retrieves its top 50 chunks through hybrid search, and the pooled results are reranked together down to the top 15.

This is where retrieval is moving. A model drives the search, deciding what to look for next and when it has enough. That used to be too slow and expensive for every query, but models are getting better, cheaper, and faster.

Agentic grep, large model. gpt-6.1-sol with a single tool: full-text and regex search over the chunks. No embeddings at all, the way a coding agent explores a repository.

Agentic grep, small model. The same setup with gpt-6.1-luna, a smaller and faster model, to see how much the strength of the model matters.

Default mode. Kapa’s faster mode: a fixed pipeline with a small planning step and a strict budget, returning a fixed number of the best-ranked chunks.

Deep mode. Kapa’s retrieval agent. It searches, reads on in documents it has found, and prunes what turns out not to be relevant, returning as many chunks as the query needs.

To compare the retrievers, we look at them across four dimensions: retrieval score, latency, precision, and how many tokens they return. A retriever that scores well can still be too slow to put in front of a user, or return so much that the agent reading it pays for it on every call.