AI search explained: indexing, retrieval and reranking

Every organisation has its data spread over multiple systems, sometimes even in vast data lakes. For example, support agents log tickets, sales teams update the CRM, engineers file bugs in the bug tracker, and most of the conversational context ends up in Slack threads and emails. Each tool is good at its own job, so over time what the organisation knows, and more importantly what it needs to make business-critical decisions, is spread across all of them.

Let's visualise the need for a central organisational knowledge search using a few synthetic examples. Say a product manager at a bike company wants to know why customers are asking for refunds. The answer is already somewhere in the data. For example, a support ticket might have details about a brake cable that snapped, an email could mention a battery that dies after 10 km, and a Slack message might include conversational details about three customers wanting their money back over faulty brakes. But this data is spread across three tools, written by three different people, and not in the same words.

The obvious first step is to search for "refund". The search will most probably return the refund policy, the refund how-to and the email, all of which include the word "refund". It will most certainly miss the ticket and the Slack message, which describe the problem in different words. The manager gets an answer that looks complete but isn't.

The problem isn't that the data is missing or incomplete. The problem is that search looks for word matches, and people often describe the same thing using different words. For an organisation to make business-critical decisions based on data it already has, search has to work across every source and understand what a query means instead of just looking for word matches.

Data is connected from many sources. Indexing runs when there's new data. Retrieval runs when a user searches.

This post walks through three types of retrieval using the same dataset of ten notes, shows where each approach fails and builds up to what a good retrieval strategy looks like:

For each method we'll look at how the data goes through two paths of the search system:

We have a synthetic dataset representing ten brief notes from eight different tools. Each one has an id, a source, some text and the basic metadata most connectors would provide:

{ "id": 10, "source": "Slack", "text": "Three customers wanted their money back this week because of faulty brakes. …" }

1WikiPolicy: refunds are allowed up to 30 days after delivery. The bike must be unused and in its original box.

My rear brake cable snapped on the second ride. I don't feel safe riding it, so I'm returning the bike.

A customer asked for a refund because the battery dies after 10 km, far short of the 60 km we advertise.

Heads up: the warehouse has run out of medium frames, so new orders will ship two weeks late.

Planning meeting: launch the cargo bike in spring and hire two mechanics for the Sydney store.

Acme Couriers renewed their contract for 40 bikes and asked for a bigger volume discount.

Survey results: riders love the smooth ride, but many say the app keeps losing its connection to the bike.

Bug: the app crashes when pairing with a bike over Bluetooth on Android 15.

How to issue a refund: open the order, click Refund and send it to the original payment method.

Three customers wanted their money back this week because of faulty brakes. Pausing Trail 2 sales until the supplier sends a fix.

The dataset includes ten notes from eight different tools.

From the dataset, three notes are relevant to our question:

Of these three notes, only note 3 contains the word "refund". Notes 1 and 9 contain it too, but they only mention the word "refund" (refund policy and refund how-to) and not a reason for a refund. This is the detail we will look for in each search to measure its effectiveness.

Text search matches the exact words in the query to the index. It's what Postgres full-text search, Elasticsearch and Lucene do underneath, and it powers most search boxes on the internet.

Indexing. A connector posts each document to the search API. The API then runs the text through an analyzer, which splits it into tokens, converts them to lowercase, drops stop words like "the" and "so", and does some lexical processing. Each processed term is inserted into an inverted index, which is a map from the indexed word to a posting list including the ids of every document that contains it. For example, adding note 10 to the inverted index would append 10 to the posting lists for terms "money", "back", "brake" and the rest.

Retrieval. The query goes through the same analyzer that's used for indexing, so "Brakes" and "brake" become the same term. The API fetches the posting list for each term, merges them and ranks the documents. This demo ranks by how many query terms a document matches. Production engines use more sophisticated ranking functions such as BM25, which weights rare terms above common ones and adjusts for document length.

Runs when a document is created or changed.

{ "id": 10, "source": "Slack", "text": "Three customers wanted their money back…" }

["three","customer","wanted","money","back","week"] …10 more

money → [10], back → [10], brake → [2, 10], …

{ "id": 10, "source": "Slack", "text": "Three customers wanted their money back…" }

["three","customer","wanted","money","back","week"] …10 more

money → [10], back → [10], brake → [2, 10], …

5. Inverted index → Search API (response)

[{ "id": 10, "source": "Slack", "text": "Three customers wanted their mon…" }]

5. Inverted index → Search API (response)

[{ "id": 10, "source": "Slack", "text": "Three customers wanted their mon…" }]

The same analyzer is used for both writes and reads, so documents and queries use the same terms.

Build the index one document at a time, then search it.

The index is empty. Press play to watch each document get split into words and filed into the index.

Try "refund", "money back" and "equipment failures".

Word matches are not enough: Text search is fast and easy to debug, but a document matches only when it contains the exact word. This is a good retrieval system for IDs, product codes, error messages and names but falls short for data with paraphrased natural language. In our example, refund returns notes 1, 3 and 9: the policy, the battery complaint and the how-to. Money back means the same thing but returns only note 10. Note 2, the snapped brake cable, won't show up in either search. And equipment failures doesn't match anything, even though all three notes describe exactly that using different words. Synonym lists can help match individual cases, but then you end up maintaining a thesaurus for every way people describe a problem, and you'll still miss paraphrases and have to deal with false positives from loose synonyms.

Embedding search keeps both paths and the same API. However, it swaps two components where the analyzer becomes an embedding model and the inverted index becomes a vector database.

Indexing. The API sends each document's text to the embedding model, which returns a vector: a fixed-length array of floats. We use bge-small-en-v1.5 in this example, which returns 384 of them. Embedding models are trained in a way that texts with similar meanings produce vectors that point in similar directions, whether or not they share words. The API upserts the vector into the vector database with the document's id and metadata, so results can come back as text and don't need a second lookup.

Retrieval. The query goes through the same model. We query the vector database, and it returns the stored vectors nearest to the query vector by cosine similarity, along with their stored metadata. With ten documents the query can compare against all of them. With millions, vector databases use an approximate nearest neighbour index such as HNSW to return the results in milliseconds. It's a hard requirement to use the same model for creating the vectors and querying them, since vectors from different models live in different spaces and can't be compared. So changing the embedding model would mean re-embedding everything.

Runs when a document is created or changed.

{ "id": 10, "source": "Slack", "text": "Three customers wanted their money back…" }

[-0.021, -0.042, 0.058, 0.044, …] (384 floats)

{ "id": 10, "vector": […], "metadata": { "source", "text" } }

{ "id": 10, "source": "Slack", "text": "Three customers wanted their money back…" }

3. Embedding model → Search API (response)

[-0.021, -0.042, 0.058, 0.044, …] (384 floats)

{ "id": 10, "vector": […], "metadata": { "source", "text" } }

5. Vector database → Search API (response)

[-0.083, -0.013, 0.061, -0.020, …] (384 floats)

{ "vector": […], "topK": 5, "metric": "cosine" }

[{ "id": 10, "score": 0.576 }, { "id": 2, "score": 0.565 }, { "id": 3, "score": 0.52 }, …]

[{ "id": 10, "source": "Slack", "text": "Three customers wanted their mon…" }, …4 more]

3. Embedding model → Search API (response)

[-0.083, -0.013, 0.061, -0.020, …] (384 floats)

{ "vector": […], "topK": 5, "metric": "cosine" }

5. Vector database → Search API (response)

[{ "id": 10, "score": 0.576 }, { "id": 2, "score": 0.565 }, { "id": 3, "score": 0.52 }, …]

[{ "id": 10, "source": "Slack", "text": "Three customers wanted their mon…" }, …4 more]

The same API shape as text search. The analyzer becomes an embedding model, and the inverted index becomes the vector database.

The demo flattens 384 dimensions to two to help visualise the space. Dots that are close together have similar meanings.

Nothing is embedded yet. Press play to watch each document turn into numbers and land on the map.

Similar meanings land close together: refunds and returns on one side, the app on another.

Embed each document into the vector database, and then query it with paraphrased words.

Embedding search is better than text search: Equipment failures, the query that text search did not find any results for, now returns the faulty brakes (note 10), the snapped cable (note 2) and the weak battery (note 3) as its top three. None of them contains either word. Stock shortages finds the warehouse out of frames, and smartphone glitches finds the app bugs.

Embedding search is not always enough: Each document is compressed into one vector during indexing. So similarity measures what a document is about instead of how relevant it is to a particular question. For example, if we ask why are customers asking for refunds?, embedding search returns:

The top result is about how to process a refund instead of why a customer might want one. The more relevant answers to the question sit below the less relevant first result. And the contract renewal has nothing to do with refunds, but it still makes the top five results. The results are meaningfully close but not relevant to what the user wants to act on.

A reranker is a second model that reorders the results. It reads the query with each result from the vector database and sorts them using a relevance score. It applies another layer of intelligence to the results of vector matches and can tell the difference between "how to issue a refund" and "why a customer wants one".

Indexing. Unchanged. Same as embedding search.

Retrieval. The read path is similar to embedding search, with an additional stage:

Unchanged from embedding search. The reranker keeps no index of its own.

["why are customers asking…", "How to issue a refund: o…"] × 5 pairs

[{ "id": 3, "score": 0.864 }, { "id": 10, "score": 0.693 }, { "id": 9, "score": 0.53 }, …]

[{ "id": 3, "source": "Email", "text": "A customer asked for a refund be…" }, …4 more]

3. Embedding model → Search API (response)

4. Vector database → Search API (response)

["why are customers asking…", "How to issue a refund: o…"] × 5 pairs

[{ "id": 3, "score": 0.864 }, { "id": 10, "score": 0.693 }, { "id": 9, "score": 0.53 }, …]

[{ "id": 3, "source": "Email", "text": "A customer asked