Needle: Search Engine Benchmark for Agents

We judge with openai/gpt-6-luna via OpenRouter, high reasoning effort, temperature 0, one pointwise call per document. We record the model for each run as judge_model in its archived report. The judgement browser shows it when you select a run, so judge changes stay visible in the history.

The judge does not see the engine identity. The prompt contains the query, the document title, the URL, the published date, and the snippet — not the engine name. We judge each distinct combination of URL, title, snippet, and date once per query and use that rating again where the identical result occurs for other engines. We judge the same URL with a different snippet separately, with the same blind judge and rubric.

A judge preference for a snippet style would be a shared bias across engines. We publish the per-result reasoning with each run, so you can audit this.

Rubric: 0–4 Needs Met scale (FailsM / SM / HM / FullyM) with hard caps for a wrong entity, stale coverage of dated events, and search-operator compliance ("quoted phrases", site:). The full prompt is one template in the repo: src/needle/shared/prompts/judgement.jinja.

Known limitation: we operate NEEDLE ourselves, and we did not yet do a systematic human-agreement audit of the judge. But you can check the numbers: the judge is blind and shared across engines, each rating includes its reasoning, and we publish the queries, prompts, and raw results. Thus anyone can score any rating again.

Every run builds each client with the same snippet budget of 2,000 characters. Keenable, Exa and Ceramic accept a request-side length parameter, so they get it in the request (snippet_max_length=2000, highlights.maxCharacters=2000, maxDescriptionLength=2000). The other engines have no such parameter; we clip whatever they return to the same 2,000 characters before judging and answer matching (Appendix C). Requests time out at 30 s (60 s for Perplexity, Parallel and Jina, 180 s for claude-search and chatgpt-search) and retry up to 3 times with exponential backoff on transport errors, 429, and 5xx.