Why king – man + woman isn't queen

Why king - man + woman isn't queen, and seven other things we measured.

An embedding is a fixed-length list of numbers that a model produces for a piece of content, so that content with similar meaning gets vectors pointing in similar directions. Vector search embeds your query with the same model and returns the stored items whose vectors are closest - usually by cosine similarity, the cosine of the angle between two vectors.

That paragraph is accurate, and it's roughly where most introductions stop. The trouble is that it leaves you with intuitions that are mostly right, and the "mostly" is where search systems go wrong in production.

So this post takes a different route to understanding embeddings: it starts from the definition above and then goes through eight surprises that show up as soon as you actually measure. Each one corrects a common intuition, and together they're a much better mental model of what an embedding is than any definition. Let's go over them:

All numbers below come from Qwen3-Embedding-0.6B, a small open-weights model run locally on a laptop, unless stated otherwise. Other modern models differ in the exact numbers, not in the shape of the results.

Text goes in, a tokenizer cuts it into sub-word pieces, a transformer turns the pieces into context-aware vectors, and a pooling step collapses those into one vector - 1,024 numbers for this model - normalized to length 1:

vec = model.encode("The warranty covers the frame for five years.") vec.shape # (1024,) float(vec @ vec) # 1.0 - unit length, so a dot product is cosine similarity

The model learned to produce these through contrastive training: pairs of texts that belong together (a question and its answer, a title and its article) are pulled together, everything else in the batch is pushed apart. Keep that training objective in mind - almost every surprise below follows from it.

The most famous fact about embeddings is that you can do arithmetic with meaning: take the vector for "king", subtract "man", add "woman", and you land on "queen". It's in nearly every introduction.

Let's try it, on a 40-word vocabulary that includes royalty, family words, capitals and some distractors:

target = E["king"] - E["man"] + E["woman"] nearest(target)

RankWordCosine1woman0.8232king0.8043queen0.7874wife0.786

"Queen" is third. The nearest word to king - man + woman is woman, followed by king.

The trick in the original demonstrations is that the input words are excluded from the candidates - word2vec's analogy evaluation and gensim's most_similar both do this. Exclude them and the answer is queen, at 0.787, just ahead of wife at 0.786. Nissim, van Noord and van der Goot wrote a whole paper on this in 2020 ("Fair is Better than Sensational"), mostly because some of the more alarming headlines about biased analogies relied on the same exclusion.

The capital-city version behaves better - paris - france + germany gives berlin at 0.907 with the inputs included - so the effect is real. But the lesson is the right mental model for embeddings: the arithmetic moves you a little in a direction; it doesn't teleport you to a concept. Most of what's in a vector is "what this is about", and subtracting man doesn't remove much of it.

Take ten sentences with nothing to do with each other - cats, quarterly revenue, a train timetable, photosynthesis, a VPN reset, a cake recipe, Everest, an API, a cellist, a bike warranty - and compute all 45 pairwise similarities:

If vectors were spread uniformly in 1,024 dimensions, unrelated pairs would sit at 0 with a standard deviation of about 1/√1024 ≈ 0.03 - with random vectors, 99% of pairs land within ±0.08. Real models don't use the whole space. Their vectors live in a narrow cone, so everything is somewhat similar to everything else. Researchers call this anisotropy, and how narrow the cone is varies by model: developers using OpenAI's older ada-002 reported unrelated texts scoring above 0.7.

It's not an artifact of those ten sentences either: across SciFact's 5,183 scientific abstracts, two random abstracts average 0.26, while a claim and an abstract judged relevant to it average 0.65 - and 95% of relevant pairs score below 0.80.

The practical consequence: a cosine similarity is not a percentage. 0.25 doesn't mean "25% similar", and a threshold like "only return results above 0.8" means completely different things on different models. Scores are only meaningful relative to other scores from the same model.

If you need a threshold, calibrate it per model: embed a few hundred real queries, look at the similarity of the right answer versus the best wrong answer, and pick a cut from that distribution. And re-do it when you change models.

"Approved" and "not approved" are closer than two sentences about cats lying on things. This one follows straight from the training objective: training pairs are about relevance, and "the refund was not approved" is extremely relevant to a question about whether the refund was approved. It's the right document to retrieve - it's just a terrible document to trust if you only check that it matched.

Embeddings are topic detectors first. Negation, polarity and quantity ("approved" vs "rejected", "5" vs "50") are faint signals on top.

Here's how the tokenizer sees an invoice number:

To check this wasn't a fluke, here's a public benchmark - SciFact, 5,183 scientific abstracts - with 200 extra queries that are just an identifier found in one document (things like bcl-214), in nDCG@10 where 1.0 means the right document always comes first:

A 13x larger model gets a lot better at questions and barely better at identifiers. That's the signature of a structural limitation rather than a quality gap, and it lines up with a 2025 Google DeepMind paper (Weller et al., "On the Theoretical Limitations of Embedding-Based Retrieval") that proves a single vector of a given dimension can only represent so many distinct top-k result sets - and shows strong 7-8B embedding models scoring under 20% recall@100 on a dataset built to hit that limit, where BM25 gets 93.6%.

An embedding of a passage is a summary of the whole passage. So what happens when the one sentence you care about is surrounded by others? Take "The warranty covers the frame for five years.", pad it with unrelated sentences from the same imaginary shop (opening hours, parking, gift cards), and measure its similarity to "How long is the frame warranty?":

Twelve filler sentences on their own, with no warranty sentence, score 0.54. With 48 filler sentences around it, the passage that contains the exact answer is only 0.05 above that - barely distinguishable from a passage with nothing relevant in it. This is why chunking - splitting documents into passages before embedding them - matters so much.

And there's a sharper version. Every model has a maximum input length, and most libraries truncate silently. Here's a 1,021-token terms-and-conditions document with the warranty clause at the very end:

At 512 tokens the clause is cut off, and the document scores below the boilerplate on its own (0.32). No error, no warning. Plenty of popular models stop at 512 tokens, and some well-known sentence-transformers models at 256.

Check the max_seq_length (or equivalent) of whatever model you use, and check what your library does past it. "Truncates silently" is the common default.

Search for "how long is the frame warranty" against an answer and a near-copy of the question:

Similarity is symmetric, and the text most similar to a question is... a similar question. That's why many models want you to mark queries and documents differently - Qwen3-Embedding takes an instruction prefix on queries, E5 models want query: and passage:, Cohere and Voyage take an input type. The instruction narrows the gap (0.87 vs 0.81 instead of 0.98 vs 0.84) but, here, doesn't close it.

In practice this means FAQ pages, forum threads and ticket titles that repeat a question tend to outrank the document that answers it. Forget the prefix and it gets worse.

Modern embedding models are multilingual, which is genuinely magical:

Spanish, French and Hebrew translations all clearly beat the unrelated sentence. But "Where is the bus stop?" - a different question in English - beats the exact French translation. Embedding spaces are multilingual, but language is still one of the directions in them, and it's not a small one. For a mixed-language corpus that means same-language results get a head start.

It's tempting to think of a vector as a one-way hash of the text - 1,024 opaque floats, surely you can't get the words back. You can, to a large extent. Morris et al.'s vec2text (2023) reconstructs 92% of 32-token inputs exactly from their embeddings (for the encoder it was trained to invert), and recovered names from clinical notes. If the text is sensitive, the vectors are sensitive too: they need the same access controls, the same retention and the same deletion.

Putting the eight together gives a better definition than the one at the top:

An embedding is a lossy summary of what a piece of text is about, in a space where everything is somewhat similar, topic dominates, and exact details - identifiers, negation, the end of long documents - fade or disappear.

That's not a knock on embeddings. They're the reason a search for "sales rose by twelve percent" can find "revenue grew 12%" (cosine 0.84, with "grew 12%" and "rose by twelve percent" sharing no word). It's an argument for using them for what they're good at, and for measuring the rest.

We ran into most of these while building search for Deployed, and they shaped how it works.

If you take one thing from this post: before you trust a vector search, embed twenty of your own queries and look at the raw numbers - hopefully with fewer surprises. Thanks for reading and hope you enjoyed!

An embedding is a fixed-length list of numbers that a model produces for a piece of content, so that content with similar meaning gets vectors pointing in similar directions. A tokenizer cuts the text into sub-word pieces, a transformer turns them into context-aware vectors, and a pooling step collapses those into one vector - 1,024 numbers for Qwen3-Embedding-0.6B - normalized to length 1.

Vector search embeds the query with the same model that embedded the documents and returns the stored items whose vectors are closest, usually by cosine similarity - the cosine of the angle between two vectors. Embedding models are trained to pull related pairs together, such as a question and its answer, so vector search finds text that shares meaning but not words: "sales rose by twelve percent" matches "revenue grew 12%" at 0.84.

There isn't a universal one. With Qwen3-Embedding-0.6B, unrelated sentences scored 0.12 to 0.37, and developers using OpenAI's older ada-002 reported unrelated texts above 0.7. To set a threshold, embed a few hundred real queries, compare the similarity of the right answer with that of the best wrong answer, pick a cut from that distribution - and redo it when you change models.

Only if you leave out the input words. On a 40-word vocabulary, the nearest word to king - man + woman was "woman" (0.823), then "king" (0.804), then "queen" (0.787). word2vec's analogy evaluation and gensim's most_similar exclude the input words, which is how the famous demonstration lands on queen. The arithmetic moves a vector a little in a direction; it doesn't jump to a concept.

A tokenizer splits an identifier like INV-20931 into seven tokens, and the embedding summarizes them into the same 1,024 numbers as everything else, so nearly identical IDs look alike: a search for INV-20931 returned INV-20913 first. On identifier queries over SciFact, BM25 keyword search scored 0.93 nDCG@10, against 0.21 to 0.30 for three embedding models - which is why search systems pair the two.

An embedding summarizes the whole passage, so every sentence that isn't the answer dilutes it: a warranty sentence scored 0.78 a