2026.07.26 // LLM FOR DUMMIES // 8 MIN
LLM for Dummies #2 — Embeddings: Meaning Is a Place, Not a Definition
Part 1 left us with a list of opaque numbers. Now the magic starts: every token becomes a point in a space with thousands of dimensions, where distance is similarity and directions are concepts. This is the idea powering semantic search, recommendations, RAG — and the input layer of every LLM.
ALESSIO ROCCHI ·
In part 1 we followed your text into the tokenizer and came out the other side with a list of IDs: 4211 for "␣models", 13 for the period. And we ended on a problem: those numbers are arbitrary. Row 4211 says nothing about what "models" means. Swap the IDs around and nothing changes.
This part is about the step that fixes that — the one genuinely beautiful idea in the whole pipeline: meaning as geometry.
A Dictionary With No Definitions
How do you explain to a machine what "cat" means? The dictionary approach — "small domesticated feline" — just moves the problem: now you need "small", "domesticated" and "feline" explained, and you're chasing your own tail forever.
The idea that won is different. Don't define words at all. Instead, give every word a position in space, and arrange the positions so that similar things end up close together. "Cat" doesn't need a definition if it sits next to "dog" and "kitten", far from "carburetor". The position is the meaning.
FIG. 01 // MEANING AS GEOMETRY
Words as points: neighbors mean similar things
ILLUSTRATION: EMBEDDING SPACE (POSITIONS NOT TO SCALE)
Concretely, every token gets a vector: a long list of numbers — thousands of coordinates in a modern model. You can think of each coordinate as a score on some hidden property. As if the model rated every word on twelve thousand unnamed scales: a bit of "animal-ness", a dash of "royalty", some "concreteness", a pinch of "Frenchness". Nobody chose those scales, and (spoiler for the deep dive) they don't map cleanly onto human concepts. But the effect is exactly that: words become describable by numbers, and comparable by distance.
Directions Are Concepts
The figure above hides a second, deeper trick. Look at the two arrows: the displacement from "king" to "queen" is the same as the displacement from "man" to "woman". A consistent direction in the space encodes the concept "gender". Another direction encodes plural. Another one, country→capital.
Which means you can do arithmetic on meaning:
king − man + woman ≈ queen
Paris − France + Italy ≈ Rome
Take the vector for "king", subtract "man", add "woman", and the nearest point to where you land is "queen". Nobody programmed this. It fell out of training — and when it was first discovered (word2vec, 2013), it was the moment much of the field realized these representations were capturing something real.
Where Do the Coordinates Come From?
Nobody sits down and assigns positions to 100,000 tokens. The positions are learned, and the principle behind the learning is old and elegant, from 1950s linguistics: "You shall know a word by the company it keeps."
Words that appear in similar contexts get nudged toward each other; words that never share a context drift apart. Do that over trillions of words and clusters emerge: animals near animals, verbs near verbs, "cat" near "dog" — not because anyone said they're similar, but because they both chase mice, sleep on sofas and knock things off tables, in text. Meaning emerges from statistics of use.
FIG. 02 // THE LOOKUP
From opaque ID to a point in space
ROW 4211 OF THE EMBEDDING TABLE
[ 0.12, −0.87, 0.44, 1.03, −0.31, … ] … × 12,288 coordinates
ILLUSTRATION: EMBEDDING LOOKUP · E ∈ ℝ^(|V|×d)
Mechanically, it's almost anticlimactic: the model owns a giant table with one row per vocabulary entry. Processing token 4211 means fetching row 4211. That's the entire "reading" process — the tokenizer picks the row numbers, the embedding table turns them into geometry, and everything the model does downstream is math on those vectors.
You Use This Every Day
The reason this part of the series matters beyond LLMs: embeddings escaped the language model and became infrastructure.
- Semantic search: you type "car", results mention "automobile" — the query and the documents are embedded, and the engine returns nearby documents, not matching strings.
- Recommendations: "people who liked this also liked…" is nearest-neighbor search in an embedding space of movies, songs, products.
- RAG (retrieval-augmented generation): when a chatbot "searches your documents", your files were embedded into vectors, and the relevant chunks are found by distance before being handed to the LLM.
- Multilingual magic: in a well-trained multilingual space, "dog", "cane" and "Hund" land close together. Translation stops being a rule book and becomes a short walk.
FIG. 03 // ORDERS OF MAGNITUDE
Embeddings in three numbers
Coordinates per token in a frontier model
~12K
the 'model width' d — every token becomes a vector this long
Cosine similarity: cat↔dog vs cat↔carburetor
0.86 vs 0.09
similarity is measurable, not metaphorical (illustrative values)
Angle between two random directions in high dimensions
≈ 90°
near-orthogonality is why one space can host millions of concepts
TYPICAL VALUES: TRANSFORMER EMBEDDING LAYERS · 2026
One honest limitation before the math: a static table gives "bank" one row — but a river bank and a savings bank deserve different points. The table alone can't fix that: the same ID always fetches the same vector. What fixes it is the next stage, where vectors get to look at each other and reshape themselves based on context. That's attention — but let's earn it first.
Deep Dive: The Geometry, Formally
The "for dummies" part ends here — below assumes comfort with linear algebra and probability. See you at part #3 otherwise: you already have the picture that matters.
The Embedding Layer Is a Matrix
The table from FIG. 02 is a matrix E ∈ ℝ^{|V|×d}, with |V| ≈ 100K vocabulary entries and d the model width (12,288 for GPT-3-class models). Token i maps to row eᵢ = E[i,:] — equivalently, E multiplied by a one-hot vector, which is why the layer is differentiable and trainable by backprop like everything else. In many architectures the same matrix is reused, transposed, as the output layer that scores the next token (weight tying): the model predicts by measuring the dot product between its final hidden state and every row of E — geometry in, geometry out.
The input to the Transformer proper is xᵢ = eᵢ + pᵢ, where pᵢ injects the position of the token (sinusoidal in the 2017 original, rotary/RoPE in most modern models — we'll treat positional encodings properly in part #3, where they matter).
Similarity: Why Cosine
The standard similarity between vectors u, v:
cos(u, v) = ⟨u, v⟩ / (‖u‖ · ‖v‖)
Why angle rather than Euclidean distance? Because in learned spaces the norm carries frequency-ish information — frequent tokens tend to grow longer vectors — while the direction carries most of the semantics. Cosine strips the norm. In retrieval systems everything is L2-normalized so cosine and dot product coincide, and "nearest neighbor" search reduces to maximum inner product — the operation vector databases optimize.
A wrinkle worth knowing: trained embedding spaces are anisotropic — vectors concentrate in a narrow cone rather than spreading uniformly, so raw cosine values run high across the board and should be compared relatively, not read as absolute percentages.
What word2vec Actually Optimizes
The classic static-embedding recipe, skip-gram with negative sampling (SGNS), trains two vector sets (center v, context u) to tell real co-occurrences from fake ones:
L = Σ_(w,c) [ log σ(⟨v_w, u_c⟩) + Σ_{j=1..k} 𝔼_{c'∼Pₙ} log σ(−⟨v_w, u_{c'}⟩) ]
Maximize the sigmoid of the dot product for pairs that occur, minimize it for k sampled negatives. The remarkable theoretical result (Levy & Goldberg, 2014): at optimum, SGNS implicitly factorizes the shifted pointwise mutual information matrix,
⟨v_w, u_c⟩ ≈ PMI(w, c) − log k, PMI(w,c) = log [ P(w,c) / (P(w)P(c)) ]
"You shall know a word by the company it keeps" becomes a precise statement: embeddings are a low-rank factorization of co-occurrence statistics. The analogy arithmetic follows: relationships that show up as consistent PMI patterns (king:queen :: man:woman) become consistent vector offsets. In practice, analogy evaluation is argmax_w cos(w, king − man + woman) excluding the three query words — the parallelogram is real but noisier than the meme suggests.
Why 12,288 Dimensions Don't Get Crowded
Concentration of measure: for two random unit vectors in ℝ^d, the expected cosine is 0 with standard deviation ≈ 1/√d. At d = 12,288, two random directions are orthogonal to within ±0.009 — high-dimensional space is unfathomably roomy. This enables superposition: a model can store far more quasi-orthogonal "concept directions" than it has dimensions, at the price of a little interference — one of the central objects of study in modern interpretability research. It's also why "small" retrieval embeddings (256-1024 dims) still work: you need surprisingly few dimensions before near-orthogonality gives every concept its own corner. Modern retrieval models push this further with Matryoshka training, which packs the most important information into the vector's prefix so you can truncate it — 1024 dims for accuracy, first 256 for speed — without retraining.
From Static to Contextual
Everything above gives one vector per token type — the "bank" problem stays unsolved at the table level. In a Transformer, the embedding layer is only layer 0: what flows through the network is a sequence of vectors that each subsequent layer transforms, mixing in information from surrounding tokens. By the middle layers, the vector at the "bank" position in "river bank" and in "bank account" have moved to genuinely different places. Contextualization is not a property of the table; it's a property of the machinery on top — the attention mechanism, which is exactly part #3 of this series.
What's Next
We now have tokens as points in space, and a space where geometry means something. But the vectors are still lonely — each token's embedding knows nothing about its neighbors in the sentence. Next time: attention — the mechanism that lets every token look at every other token and decide whom to listen to, the idea the entire Transformer architecture is named after ("Attention Is All You Need"), and the reason context windows cost quadratically.
Series roadmap: tokenization → embeddings (you are here) → attention & the Transformer → training → from autocomplete to assistant → inference. Questions? Tell me.