What are vector embeddings? An interactive guide

This guide walks you through the history, design, theory, and practical applications of vector embeddings. For the first time, we bring this abstract subject to life through interactive visualizations that let you see and manipulate the invisible geometry of meaning underpinning everything from Google Search and your Twitter feed to ChatGPT.

We begin with the problem embeddings were created to solve: how to represent meaning numerically so that computers can work with it. From there, we build an intuitive understanding of semantic space, vector arithmetic, and semantic similarity.

We then put these principles to work, showing how embeddings can power real business systems through techniques such as semantic search, hierarchical retrieval, zero-shot classification, clustering, and deduplication.

No programming experience is required. For readers who want to recreate the techniques, we have included short Python snippets demonstrating how each one works.

For a taste of what lies ahead, try the GeoGuessr/Wordle-style word-trivia game below. It projects 43,339 English words onto a 2D map using real embeddings from Kanon 2 Embedder, then selects one as the target. Enter words or vector expressions to place markers, then use them to trace the smallest possible loop around the target word. We explain how embeddings power the map and its mechanics later in the guide; for now, let the spirit of exploration guide you as you take your first plunge into the hidden geometry of language.

king − man + woman

• − × ÷ min max avg

Introduction

Vector embeddings are among the most common and versatile tools in modern natural language processing. They power everything from Google Search and Netflix recommendations to the memory and retrieval capabilities of AI chatbots. At their core, embeddings allow computers to work with not only the characters in a piece of text, but also the meaning those characters cumulatively convey.

The underlying problem is simple. Humans intuitively understand words and sentences, while computers ultimately work with binary digits represented as 0s and 1s.

For a computer to process text, its characters must first be expressed in this binary form. In UTF-8, for example, the capital letter C is represented by the byte 01000011. Once encoded, it can be stored, copied, and processed. This translation into a machine-readable form is known as encoding.

Character encoding tells a computer which symbols appear, but not what those symbols mean. What if we could encode the meaning of words and sentences in a similar way?

An embedding model attempts this transformation at a higher level. Rather than encoding individual characters, it encodes the meaning of a word, sentence, or document as a vector, which is simply a list of numbers. The sentence a warm summer evening, for example, might become [0.18, -0.42, 0.73, ...]. Once meaning has a numerical form, computers can compare texts, search by meaning, and identify relationships that their exact wording alone cannot reveal.

Despite their importance, embeddings remain poorly understood. Generative models expose their behavior through the text they produce, while embedding models return long lists of numbers that are far more difficult to interpret directly. Most people therefore encounter embeddings indirectly, beneath search engines, recommendation systems, and AI chat services.

This lack of visibility affects engineers too. Because generative models are easy to prompt and work with, they are often chosen over faster, cheaper, and more accurate model alternatives. The recent attention around Jev, a non-generative model designed to make fast, structured decisions inside software, reflects a growing interest in alternatives to using chat models for every AI task. Embedding models are another important part of this broader non-generative toolkit.

Before we can use embeddings to automate sophisticated decision-making tasks, we need to start with the fundamentals: what embeddings are, how they work, and how they came to be.

Humans have been grappling with the relationship between symbols and meaning for as long as we have been writing.

As far back as 3000 BC, writing systems such as cuneiform and Egyptian hieroglyphs used visual forms to represent objects, sounds, and ideas. A sun could be represented by a disc 𓇳, water by a series of ripples 𓈗, and a house by its floor plan 𓉐. Other signs depicted a human being 𓀀 or the head of an ox 𓃾.

Representing a sun or an ox is relatively easy. Representing the past, fear, intention or sadness is much harder. These concepts have no physical form that can be reduced to a simple picture. So how do we represent abstract meaning without turning every symbol into a Rorschach test?

One approach is to arrange words along scales that capture part of their meaning. Temperature-related words, for example, have a natural order that we can encode numerically.

The final column shows how each value can be stored in signed 4-bit binary. The numbers are illustrative, but assigning each word a numerical position allows a computer to perform arithmetic whose results reflect our intuition about how the words relate.

This is a rudimentary one-dimensional embedding. It captures temperature, but nothing else. Warm and muggy, for example, occupy the same position despite having different meanings. Adding humidity as a second dimension would separate them, while a third could capture register, distinguishing an everyday word such as hot from a more literary word such as torrid.

Adding more dimensions would produce richer representations, but deciding what every dimension should mean and assigning every word a position by hand would quickly become impractical. Modern embeddings grew out of attempts to learn this structure automatically from how language is used.

In 1990, latent semantic analysis introduced one way to do this. Instead of manually defining properties such as temperature or humidity, it represented words according to how often they appeared across a collection of documents. Each document initially acted as a dimension, producing a large but crude map of word usage.

That map was then compressed into a smaller set of latent dimensions. Words that appeared across similar documents received similar positions, allowing relationships to emerge from patterns in the text rather than being defined by hand.

Neural language models later learned these representations directly from unlabeled text through self-supervised learning. Word2vec helped popularize this approach in 2013 by learning relationships between words from their surrounding context. Modern embedding models extend the same principle to phrases, sentences, and entire documents.

Unlike our temperature example, the dimensions learned by these models rarely correspond to a single property that we can name. Meaning is distributed across the vector as a whole. Taken together, a few hundred dimensions can capture distinctions as subtle as differences in legal wording between jurisdictions or the passive aggression buried in a midday text from your partner.

The process of converting plain text into a vector embedding is known as vectorization. Modern embedding models such as Isaacus’ Kanon 2 Embedder can vectorize anything from a single word to an entire document in milliseconds.

Kanon 2 represents every input as a fixed-length list of 1,792 numbers. Each number is a coordinate along one of the model’s learned dimensions, giving every embedding a dimensionality of 1,792. Together, these coordinates determine the text’s location within a high-dimensional vector space, where passages with similar meanings tend to appear closer together.

To see this structure in practice, the visualization below projects a small set of word embeddings into three dimensions. Each point represents a word, while the distances and directions between points reflect how the model relates them.

→ Model Kanon 2 Embedder

→ Vector dimensions 1,792 dimensions [ −0.007+0.027−0.001+0.031−0.020−0.007−0.018+0.010 … ]

3D 2D

?

Reducing 1,792 dimensions to three inevitably loses information and distorts the exact distances between points. Even so, enough of the original structure remains for several relationships to be visible.

The horizontal displacement from king to queen follows roughly the same direction as the displacement from male to female. This suggests that the model represents the distinction between these gendered concepts consistently.

Switching the visualization to 2D makes a second relationship easier to see. Once depth is removed, the horizontal and vertical displacement from English to French resembles that from London to Paris. In this projection, one direction has come to encode a separation between classes of cities and nationalities, while another preserves the distinction between English-associated concepts and French-associated ones.

An embedding represents each word as a point in semantic space, or equivalently as a vector from the origin to that point. Subtracting two word vectors produces another vector: the displacement between them. A word vector tells us where a concept is, while a displacement vector describes how meaning changes as we move from one concept to another.

Because word and displacement vectors share the same coordinate space, we can apply a relationship found between one pair of concepts to another. Subtracting man from king, for example, captures the shift from a man to the gender-neutral concept of a monarch. Adding woman to that displacement vector reintroduces gender and produces a vector close to queen.

To build a better intuition for vector arithmetic, try some of the following expressions in the interactive visualization below:

?

For most arithmetic operations, the results will not always resolve as neatly as king − man + woman. Embedding arithmetic reflects statistical patterns learned from language rather than a fixed system of symbolic rules.

Nevertheless, using vector arithmetic through an exercise like this is helpful for understanding what knowledge embedding models are potentially encoding.

So far, we have seen how embeddings map words and phrases into a high-dimensional semantic space, and how vector arithmetic can reveal relationships within it. The embeddings themselves, however, are still lists of 1,792 numbers. Computers can process these lists easily, but they are largely unintelligible to humans. To make practical decisions with them, we need to reduce the relationship between two vectors to a single score.

Linear algebra gives us several ways to make this comparison. Each examines a different aspect of the vectors’ geometry and returns a scalar, which is simply a single number. We can then use this value to estimate semantic similarity, or how closely two pieces of text are related in meaning. Estimates of semantic similarity include:

Cosine similarity is commonly used for text retrieval because it compares semantic direction while normalizing away differences in vector length that may be unrelated to relevance. Mathematically, it is written as

The visualization below shows how cosine similarity turns this geometry into a score. Choose a reference sentence to compare its vector with the other examples. The more closely two sentences are related in meaning, the more closely their vectors align, producing a smaller angle and a higher similarity score.

3D 2D

?

With ‘A bat is flying out of a cave’ selected as the reference, ‘A flying fox slipped out of the grotto’ receives the highest similarity score of the remaining sentences at 0.55. A flying fox is a type of bat, while a grotto is similar to a cave. The model recognizes this relationship despite the differences in wording.

‘His bat went flying from his hands’ shares the words bat and flying with the reference, but scores lower at 0.48 because bat now refers to sporting equipment. When this sentence be