Building a reinforcement learning system for e-commerce search

Someone types "winter gloves heated" into Agora.

It is a short query, but answering it well takes several decisions. Does this person want battery-heated gloves, or just warm ones? Which of the thousands of glove listings across our merchants actually have a heating element? Which ones are replacement batteries that happen to say "gloves" in the title? And of the real matches, which ones should come first?

Agora searches millions of listings from thousands of independent stores. Every merchant writes titles differently, and shoppers write queries differently again. We could not hand-tune our way to good results across all of that. So we built a search system that learns from its own traffic.

Once a day, it takes the last 24 hours of searches, compares what it returned with what Amazon returned for the same queries, works out which stage of the pipeline fell short, and trains new versions of the models that interpret queries and rank products. Tomorrow's searches run on the new models and produce the data for the next round.

This post explains how that loop works, what we mean (and do not mean) when we call it reinforcement learning, and why we think it compounds as more stores and shoppers join.

A search goes through four stages before anything reaches the shopper:

Planning. A fine-tuned Qwen 2.5 model turns the shopper's words into a structured search plan.

Retrieval. Typesense pulls 100 candidates using keyword and vector search together.

Reranking. A cross-encoder scores every candidate against the request and keeps the best.

Checks. Deterministic code verifies the models' output against data we own.

The models in this path are self-hosted and served through Ollama. The one exception is our deep research mode, used for gift ideas and use-case questions, which calls Claude to write longer summaries and buying guides. Even there, the candidates come from the same retrieval and ranking pipeline.

Self-hosting matters for the loop. We can retrain the models every day, the search logs never leave our infrastructure, and the cost of a search does not depend on someone else's API pricing.

The planner reads the query and produces something a search engine can run. For the gloves example, the plan looks roughly like this (simplified):

{ "resolvedQuery": "heated gloves", "embeddingQuery": "heated winter gloves battery powered", "searchTypes": ["product_type"], "filters": { "categories": ["gloves"] }, "identifiers": [], "sort": null }

There are two queries, not one. resolvedQuery goes to keyword search and should only contain words that are likely to appear in a product title. embeddingQuery goes to vector search and can carry context that titles never state.

This split matters more than it looks. If a shopper asks for sandals that go with a tan dress, "tan dress" belongs in the semantic query and the reranker, not the keyword query. Otherwise keyword search will happily return dresses.

The search type also matters. It sets how much retrieval leans on keywords versus vectors. Exact-product and compatibility queries ("iPhone 15 case") lean lexical. Gift and use-case queries lean semantic. Generic product types sit in between.

Our index runs on Typesense across ten shards. Each listing has a 256-dimensional embedding from Qwen3-VL, built from its image and text, so a text query and a product photo live in the same vector space.

Retrieval runs keyword and vector search in one hybrid query. Each shard returns 20 hits, and they merge into a pool of 100. Hard constraints like price, category and rating are applied as filters in Typesense, not left to a model to remember. If the shopper named a store or brand, a second search runs against that catalog alone, so the store's products reach the pool even when their titles do not repeat the store's name.

The reranker is a cross-encoder based on Qwen3-Reranker-4B. The embedding model encodes the query and each product separately. A cross-encoder reads them together and returns a relevance score between 0 and 1. That is slower, but much better at the distinctions shopping depends on: a heated glove versus an insulated one described as "extra warm," or the glove versus its replacement battery.

Candidates scoring below 0.4 are dropped. If the shopper named a store or brand, its listings move up, but only when they score within 0.15 of the best result on the page. A wrong item from the right store should not outrank a strong match from somewhere else.

If fewer than three candidates survive, the planner gets one more attempt. It rewrites the plan using the failed candidates as evidence, retrieval runs again, and the stronger of the two pools wins.

On staging, retrieval takes about 150 ms and reranking about 1.5 seconds. The language model calls around them take longer, and a full conversational turn currently takes several seconds. That is the main cost of this design, and we are working on it.

Small self-hosted models make mistakes. We decided early not to patch those mistakes with phrase lists or regexes over the shopper's words. Rules like that grow forever and break quietly. Instead, we check each model output against data the system already owns.

Invented constraints. A price filter is accepted only with evidence: a number in the message, a remembered budget, or the limit of the search being refined. Otherwise the planner made it up, so we drop it and log it.

Misread products. If the planner decides the shopper named a product type, we check that word against the catalog. In product titles, the type usually comes last ("Baffin Tundra Boots") and brands and modifiers come first. A word counts as a product type only if it is the head noun in at least 30% of the titles that contain it. "Boots" passes. "Fine," from the typo "fine me coats," does not.

Missing products. If the shopper named a product and none of the returned listings is that product, we search for it directly and keep whichever page has it.

These checks protect the shopper. They also protect the training data. A system that learns from its own logs will learn its own mistakes, unless something catches them first.

Serving produces logs. Learning turns logs into new models.

For each search, we record the query, the plan, the pool of 100 candidates, and the ranked page. Then we run the same query on Amazon and store its top results next to ours. Amazon's products are embedded into the same vector space as our catalog, so we can compare the two result sets even when they share no listings.

From that comparison we compute a few signals:

Coverage: how many of the benchmark's products, or close equivalents, appear in our results.

Visual overlap: how many of our results sit near the benchmark's results in embedding space.

Title overlap: how much the two result sets agree lexically.

Missed lookalikes: products that were in our pool, closely matched the benchmark, and were ranked out anyway.

Each signal is computed twice: once on the final page and once on the whole pool. That is the most useful part of the record, because it tells us which stage failed.

Good matches in the pool, not on the page

No good matches in the pool, but comparable products in our index

A concrete example. In September we pulled a set of 65 searches where Amazon's results looked better than ours. Across that set, our candidate pools averaged 19.5% visual overlap with Amazon's results. Our final pages averaged 6.2%. In 48 of the 65 searches, the pool scored better than the page. The right kind of products were there, and the reranker was not choosing them. That is a ranking problem, and it is the cheapest kind to fix.

To be clear, that set was chosen because it failed. Those numbers say nothing about our average quality. They show where the headroom was.

Once a day, the last 24 hours of logs become training data for new versions of the planner and the reranker.

Retraining on logs has a subtle failure mode. A model retrained only on old logs will forget any output field the old logs did not contain. When we add a new field to the planner, such as a shopper's standing requirements or a relative price ("cheaper than that one"), the training rows need to be labeled with it, or the next model quietly unlearns it. That labeling work is in progress.

The harder question is not whether we can produce a new model every day. It is whether each one is better than the last. [CONFIRM: describe how a new version is evaluated before it serves traffic, for example a fixed held-out query set and multi-turn fixtures it was not trained on, plus what counts as a regression.]

Someone will ask, so we should answer directly.

In the textbook sense, reinforcement learning means an agent takes actions, the environment returns a reward, and the agent updates its policy to earn more reward. Our loop has that shape. The planner and reranker are the policy. Each search is an action. The comparison with the benchmark, and the diagnosis of which stage failed, is the reward signal. Each day the policy updates, and the new policy generates the next day's data.

What it is not, today, is online RL. Models do not update during a session. We do not explore by deliberately showing worse results to see what happens. The update step itself looks a lot like supervised fine-tuning and preference training on logged data. A more precise description would be batch, off-policy learning from a weak external reward.

That comes with known problems. Logs only show outcomes for what the previous model chose, not for everything it could have chosen. And a reward based on another marketplace can, at best, teach us to look like that marketplace.

We call it reinforcement learning because the loop is the important part: the system's own behavior produces the signal that improves it, every day, without anyone writing new rules. But we would rather be precise about the mechanism than borrow a term that suggests more.

It may seem odd for us to benchmark against Amazon. In one test of 50 product searches, Consumers' Checkbook found that 54% of the first 25 Amazon results were paid placements. We are not trying to copy that.

But Amazon has something a young search engine lacks: years of shopper behavior across an enormous range of queries. Its results contain a lot of learned knowledge about what people mean when they type "winter gloves heated." That makes it a useful reference for query understanding, as long as we remember what else is in it.

So we treat it as evidence, not ground truth. Disagreeing with Amazon is not automatically a mistake. Its catalog is not ours. Its ranking reflects its own incentives. Two different products can meet the same need equally well. What we learn from a comparison is not "rank like Amazon." It is "our reranker threw away a heated glove that was sitting in the pool."

The longer-term plan is to replace this teacher with a better one. The bottom of our diagram says "future user actions" for a reason. Today the loop learns mostly from the benchmark. As usage grows, it will learn more from what shoppers actually do: what they open, compare, buy, and return. Those signals measure our goal directly instead of through someone else's results.

Most search systems get better when engineers make them better. This one is built to get better as it is used. There are three reasons.

More stores mean better candidates. A catalog gap is the one failure no model can fix. Every new merchant adds products that can enter the pool, and turns some of yesterday's catalog gaps into ranking problems the loop can solve.

More searches mean better coverage of intent. Each day's training data is made of the queries people actually typed. The long tail of shopping queries is very long, and the only way to cover it is to see it.

More shoppers mean better rewards. A benchmark comparison is a proxy. Shopper behavior is the real thing. The more of it we have, the less we depend on anyone else's idea of relevance.

The