Keyword Search 101: How BM25 Scores

The formula in five minutes, then nine things it does that you only see when you measure.

BM25 (short for Best Match 25) is the ranking function behind most keyword search: for each word of the query that a document contains, it multiplies how rare the word is across the collection (its IDF, short for inverse document frequency) by how often it appears in the document - a count that saturates and is normalized by the document's length - and adds the results up. Elasticsearch, OpenSearch, Solr, Lucene, Tantivy, Vespa and SQLite's FTS5 all rank with it by default or on request.

It's also the oldest part of most modern search stacks - published in 1994, and still the baseline every embedding model is measured against. Most explanations stop at the formula. This one goes through the formula on a five-document example, and then through nine surprises that only show up once you run it on real documents and look at the numbers. Let's go over them.

Three ideas, multiplied. Rare words matter more (IDF). The tenth mention matters less than the first (saturation, through k1). A mention in a short document says more than one in a long document (length normalization, through b).

Let's run it on an invented help centre - Fernhill Cycles, five pages - with the query frame warranty:

frame is in 2 of the 5 pages, so its IDF is ln(1 + 3.5/2.5) = 0.875; warranty is in 3, so ln(1 + 2.5/3.5) = 0.539. The average page is 19.4 words. For D1, at 9 words, the length factor is 1 - 0.75 + 0.75 × 9/19.4 = 0.598, so each of its single mentions counts 1 / (1 + 1.2 × 0.598) = 0.582:

D1 - the one page that is actually about frame warranties - wins, even though D4 says "frame" four times and D3 says "warranty" five. Keep this example in mind; three of the surprises below are visible in it already.

The setup for the rest. The measurements use the three public datasets from the chunking post - research papers (Qasper), contracts (LegalBench-RAG) and meeting transcripts (QMSum), 660 questions with the exact evidence marked - cut into fixed 512-token windows and into structure-aware units with headers. BM25 is written out in about 60 lines of numpy so that each variant is the same code with one part changed; its Lucene variant matches the bm25s library's scores to within 0.00001. The score is the share of each question's evidence inside the top 2,000 tokens of results, as in the chunking post. The corpora are small (900-3,500 chunks), so read the differences, not the levels.

The IDF in most textbooks - the Robertson/Spärck Jones weight - is ln((N - n + 0.5) / (n + 0.5)), without Lucene's 1 +. For a word in more than half the documents, that's negative.

In the Fernhill example, warranty (3 pages of 5) gets -0.336 and frame +0.336. D1 contains each once, so its two parts cancel exactly: the perfect page scores 0.000, tied with the shipping page. D3 - five mentions of "warranty" - scores -0.237, below pages that don't match at all.

That sounds like a toy problem, but it isn't one. After removing stopwords and stemming, "agreement" is in 80% of the contract windows and "any" in 88%; 99% of the contract questions contain at least one word that's in over half the chunks. Measured:

(Fixed 512-token windows; recall within 2,000 tokens.) Keeping negative weights cost up to 3.8 points, outside the noise on three of the six dataset-chunking pairs. Lucene's 1 + is there so a word can only ever help.

rank_bm25, a popular Python package, uses the textbook IDF and replaces negative values with a small positive floor (epsilon times the average IDF). If you've seen scores there change when a common word is added to the corpus, that's why.

The saturating count tf / (tf + k1) (at average length) never exceeds 1. At the default k1 = 1.2, one mention is worth 0.45 of the ceiling, two 0.63, five 0.81, twenty 0.94. That's why D3 - five "warranty"s in 39 words - only reached 0.379.

Turn saturation off - score with raw counts times IDF, which is roughly what a naive TF-IDF does - and D4 wins the Fernhill example with 3.50, D3 comes second with 2.69, and the right page drops to third. On the real datasets it's one of the two most expensive parts to remove (IDF is the other):

(Structure-aware units with headers.) Raw counts cost 7 to 20 points, outside the noise everywhere. The contracts suffer most: legal text repeats the same defined terms dozens of times, and without saturation the chunk that says "Licensee" thirty times beats the clause that answers the question.

b is the part of BM25 people tend to explain at length, so it's surprising how little it moves here. Setting b = 0 (no length normalization at all) against the default 0.75:

0.8 to 1.7 points. The reason is the chunking: length normalization exists for collections where one document is a tweet and the next is a book. Chunks of a few hundred words have already done most of its work. Index whole documents and b matters much more.

Every combination of k1 (0 to 10) and b (0 to 1), 70 settings, on all six dataset-chunking pairs:

The best settings are all over the grid, and that's exactly the point: on a few hundred questions, they are mostly noise. Even chosen on the test questions themselves - the most optimistic tuning there is - they beat the defaults by 0.4 to 1.5 points, and the interval reaches zero five times out of six. The surface is flat around the defaults: with k1 anywhere from 0.9 to 2, 70% of the settings are within 2 points of the best (the contracts, which prefer a lower k1, about half). Only k1 near 0 (counts ignored) or 10 (barely any saturation) falls far behind.

Tune k1 and b only with a few thousand judged queries and a held-out split. Below that, a tokenizer change or a better chunking moves the needle more than any parameter.

Lucene, the original Okapi papers, ATIRE, BM25L and BM25+ each write the formula slightly differently. Kamphuis et al. (2020) compared the variants on several test collections and found the differences small. Here too:

Within 1.6 points, with one exception - the negative IDF of surprise 1. BM25L and BM25+ were designed to stop very long documents from being over-penalized; over chunks, there are no very long documents, so there's nothing for them to fix.

Lucene also dropped the (k1 + 1) factor that the Okapi team's TREC-3 formula has in its numerator, in Lucene 8.0 (LUCENE-8563). It scales every score by the same constant, so it changes no ranking - but it does mean a BM25 score from Lucene 7 and one from Lucene 9 differ by a factor of 2.2 (k1 + 1, at the default k1) for the same document.

To keep the index small, Lucene doesn't store each document's exact length. It stores one byte (SmallFloat.intToByte4): lengths up to 40 are exact, and above that the length is rounded down to four significant bits. A 100-word document is stored as 96, a 300-word one as 280, a 500-word one as 472 - up to about 10% off.

Running the same encoding over these chunks:

Two thirds of the rankings change, and the quality barely moves - many top-10 scores are within a few percent of each other, so tiny length changes reorder them. If you've ever tried to reproduce Lucene's scores exactly from your own implementation and been a few percent off, this is the usual reason.

It's tempting to treat a BM25 score as a confidence: show results above 10, say "no results" below. Here's the top score each question got:

A BM25 score is a sum over query words, so it mostly measures how many words the query had: the correlation between the top score and the number of query words is 0.57 to 0.79. On the contracts, the median top score when the first result was wrong (17.5) was higher than when it was right (14.6) - long questions that name a party and a clause type match many chunks well.

There's no threshold that means "relevant" across queries, or across indexes: the IDF depends on the collection, so the same document scores differently in a bigger one. To compare or fuse BM25 with anything else, normalize per query (divide by that query's best score, or use ranks).

Every BM25 implementation starts by turning text into words, and the choices there move results more than most of the formula:

Stopwords plus stemming are worth 3.6 points over lowercasing alone on the contracts ("terminate", "terminated", "termination"), and cost 2.2 on the meetings, where the same words in passing conversation match everything. Stopwords alone cost the papers 1.7. Adding stemming to stopwords moved the contracts by 5.0 points; the best k1/b in surprise 4, by at most 1.5.

IDF needs N and n: how many documents exist, and how many contain the word. Which documents get counted is an implementation detail with visible effects:

None of these is a bug in BM25. They're a reminder that the "collection" in its formula is whatever your engine decides to count.

BM25's limits are just as clear in these numbers: it can only match the words that are there. That's why most search systems now pair it with vectors - and why it's still in them.

BM25 had more surprises in it than we expected when we built search for Deployed.