返回资讯中心
外部精选
软件工程
#开发者实时流

Jev is NOT a reranker (but it's close)

The score measures how much of each criterion the top 10 reranked chunks cover, with partial credit when only some of the required chunks are found, averaged across all 1,000 cases.

Hacker News 最新emil_sorensen2026/10/16 · 2026/10/169 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

The score measures how much of each criterion the top 10 reranked chunks cover, with partial credit when only some of the required chunks are found, averaged across all 1,000 cases.

This is a test setup, not what we run in production. We picked a standard retrieval pipeline because it makes rerankers easy to compare. For each question, an LLM splits it into three search queries. We retrieve the top 30 chunks for each query, merge them with reciprocal rank fusion, and send the top 50 to the reranker. Then we score the top 10 chunks it returns.

We tested Voyage rerank-2.5 and rerank-3, Cohere rerank-v3.5, v4.0-fast and v4.0-pro, the open-source Qwen3 reranker 8B via OpenRouter, and Jev 1.13 in two modes. We capped the candidate set at 50 because Jev needs one call per chunk, and 50 is a common candidate set size for reranking anyway.

Jev isn't trained as a reranker. It's a general-purpose decision model: you give it a state (here, the query and one chunk), a question and a set of labels, and it returns a probability distribution over those labels. To turn that into a ranking, you pick a question and a way to collapse the distribution into one number per chunk. There are two standard ways to do it, and we tried both.

Noul (binary). We ask a yes or no question: is this chunk relevant to the query? Jev returns the probability of yes and of no, and we sort the 50 chunks by the probability of yes. It's a pointwise classifier, with one call per query and chunk pair.

Score (graded). We ask how relevant the chunk is on a scale of relevance levels, from unrelated to a perfect match, each mapped to a number. Jev returns a probability for each level, and we sort by the expected score: each level's value times its probability, summed. This is the method turbopuffer used. Because it uses the whole distribution, it can tell a chunk that is certainly partly relevant apart from one that is either a perfect match or unrelated.

Voyage rerank-3 scored 0.535, at $1.91 per 1,000 questions and 0.18s median latency. No other reranker beat it on any of the three.

Jev came in just behind. At 0.519, the Noul mode beat Cohere's newest model, rerank-v4.0-pro, on score (0.506), cost ($2.49 vs $3.42) and latency (0.30s vs 0.77s). The two Jev modes came out almost even: 0.519 for Noul and 0.516 for Score, with Score costing a bit more ($2.82 per 1,000 questions). On our data, the extra detail of the graded scale didn't improve the ranking.

The open model was the most expensive. Qwen3 reranker 8B cost 4.3x more than Voyage, took 14x as long, and ranked second to last.

Every model saw the same 50 chunks, and only the order changed. Even so, the best score was 18% higher than the worst (0.535 vs 0.454).

First, results depend on the data. turbopuffer's benchmark and ours point in different directions, and both can be right. If you're choosing a reranker, test it on your own data.

Second, Jev is worth a look if you want a reranker you can steer with a rubric. For our domain, though, Voyage was the best, so we're sticking with it. Good job to Tengyu Ma and the rest of the Voyage team at MongoDB.

Cohere costs come from billed units. For Qwen3 reranker 8B we used what was available on OpenRouter. Self-hosting it would most likely make it cheaper and faster.

Give your n8n AI Agents accurate answers from your docs, tickets and wikis, with no RAG pipeline to build. Free for 14 days.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →