LLM for Dummies #3 — Attention: How Words Learn to Look at Each Other

Quick recap of the series so far. Part 1: your text becomes a sequence of token IDs. Part 2: each ID becomes a point in a space where distance is similarity and directions are concepts. And we closed on an unsolved problem: the table gives "bank" one vector, whether it sits next to "river" or next to "account". The points are lonely. Each one knows nothing about its neighbors in the sentence.

This part is about the mechanism that ends the loneliness — the single idea the modern AI boom is built on: attention.

Every Word Gets to Ask Around

Read this sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? You answered instantly: the animal. Streets aren't tired. Now change one word — "because it was too wide" — and "it" flips to the street. You resolved that by letting "it" consult the rest of the sentence.

Attention is exactly this, made mechanical. Every token gets to look at every other token, assign each one a weight — "how relevant are you to figuring out my meaning here?" — and then rebuild itself as a weighted mix of what the others offer.

FIG. 01 // WHO LOOKS AT WHOM

One token deciding whom to listen to

ILLUSTRATION: SELF-ATTENTION WEIGHTS FOR ONE HEAD

That's the whole trick. No grammar rules, no parse trees, nobody told the model what a pronoun is. Relevance weights are learned from data, and coreference resolution — one of the historically hardest problems in computational linguistics — falls out as a side effect.

Questions, Labels, Contents

How does a token decide whom to listen to? The mechanism has three ingredients with famously unhelpful names — query, key, value. Here's the version that actually sticks.

Imagine each token walks into a library:

• it carries a question it wants answered ("I'm a pronoun, who's my referent?") — the query;
• every book on the shelves has a label on its spine describing what's inside ("I'm an animate noun, subject of this sentence") — the keys;
• and each book has actual content to contribute — the values.

A token compares its question against every label. Strong match, big weight. Then it takes home a blend of the contents, weighted by those matches — and that blend becomes its new, context-aware self. "Bank" that asked around and got mostly river-related content is now, literally, a different vector than "bank" that got finance-related content. The problem from part 2 is solved.

One more twist: this doesn't happen once. Each layer runs many attention "heads" in parallel — separate sets of questions and labels, each free to specialize. In trained models you find heads that track syntax, heads that link pronouns to referents, heads that just watch the previous token. Nobody assigns these roles; they emerge.

The Machine Around It

Attention alone isn't a language model. The full recipe, distilled:

FIG. 02 // THE MACHINE

The Transformer, minus the noise

ILLUSTRATION: DECODER-ONLY TRANSFORMER (GPT-STYLE)

Two alternating moves, stacked dozens of times: attention (tokens talk to each other) and a feed-forward network (each token thinks alone — same little two-layer network applied at every position, and, counterintuitively, where most of the parameters and much of the stored "knowledge" live). Residual connections let every block add its contribution to a running stream instead of replacing it, which is what makes 100-layer stacks trainable.

This architecture is the Transformer, from the 2017 paper with the most confident title in machine learning history: "Attention Is All You Need." Its real revolution was economic as much as scientific: the previous generation of models read text sequentially, word by word, like a human. Attention processes all positions in parallel — exactly the workload GPUs are built for. Parallelism made training on trillions of words affordable, scale made the models startlingly good, and everything in this AI cycle followed from there.

The Bill: Why Context Costs Quadratic Money

There's no free lunch. "Every token looks at every token" means n tokens require n² comparisons. Double your context, quadruple the attention work.

FIG. 03 // THE QUADRATIC BILL

Attention in three numbers

The paper that started it all

2017

"Attention Is All You Need" — 8 pages, ~200K citations

Double the context, multiply the attention work by

×4

every token attends to every token: n tokens, n² pairs

Attention pairs in a full 128K-token context

~16B

per layer — this is why long context costs real money

VASWANI ET AL. 2017 · TYPICAL DECODER-ONLY VALUES

This is the deep reason behind several things you've experienced:

• Long context is a premium feature. A 128K window isn't 16× the work of an 8K window — it's up to 256× at the attention layer.
• The "language tax" from part 1 compounds. Italian's +40% tokens becomes roughly +100% attention compute for the same content. Tokenizer inefficiency gets squared.
• Long chats slow down and eat memory. The model keeps cached representations of everything (the KV cache, for the deep divers below), and it grows with every message.
• The model never "skims". Within its window, every past token is re-consulted for every new token generated. When it misses something mid-conversation, it's not because it didn't look — it's because attention weights are spread thin ("lost in the middle" is a real, measured effect).

---

Deep Dive: The Formula Everyone Quotes

The "for dummies" part ends here — below assumes linear algebra and some probability. Otherwise, see you at part #4: the mental model above is the one that matters.

Scaled Dot-Product Attention

The whole mechanism is one line:

Attention(Q, K, V) = softmax( QKᵀ / √d_k ) · V

Unpacking. The layer input is X ∈ ℝ^{n×d} (n tokens, width d). Three learned projections produce queries, keys, values: Q = XW_Q, K = XW_K, V = XW_V, with W_Q, W_K ∈ ℝ^{d×d_k}. The matrix QKᵀ ∈ ℝ^{n×n} holds every query-key dot product — the "question vs label" match scores from the library story, all n² of them at once. Softmax turns each row into a probability distribution (the arc weights of FIG. 01), and multiplying by V takes the weighted mix. That n×n matrix is the quadratic cost made flesh.

Why divide by √d_k? If the components of q and k are roughly independent with unit variance, the dot product ⟨q,k⟩ has variance d_k — scores grow like √d_k in magnitude. Feed large scores to a softmax and it saturates: one weight ≈ 1, the rest ≈ 0, gradients ≈ 0, training stalls. Dividing by √d_k keeps the score variance at 1 regardless of head size. A tiny denominator carrying the whole trainability of the architecture.

Causal masking: in a GPT-style decoder, position i must not see positions j > i (it's trying to predict them). Implementation: set those scores to −∞ before the softmax, which turns them into exact zeros after it. Training on a sequence of length n thus yields n next-token prediction problems in one parallel pass — this, as much as anything, is the Transformer's efficiency miracle.

Multi-head: split into h heads of size d_k = d/h, run attention independently in each, concatenate, project with W_O. Heads are cheap subspace projections, not h copies of the full computation — same total FLOPs as one big head, much richer behavior.

Attention Is Permutation-Blind — Hence Positional Encodings

Look at the formula again: nothing in it knows where a token sits. Shuffle the input and the outputs shuffle identically — "dog bites man" and "man bites dog" would be indistinguishable. Position must be injected explicitly.

The 2017 paper added sinusoidal vectors to the embeddings: PE(pos, 2i) = sin(pos/10000^{2i/d}), cosine for odd dimensions — a spectrum of wavelengths letting the model read absolute and, via trigonometric identities, relative offsets. Modern models mostly use RoPE (rotary embeddings): instead of adding position to the vector, rotate each 2D slice of q and k by an angle proportional to position. The elegance: a rotation by mθ on q and nθ on k makes their dot product depend only on m−n — relative position falls out of the geometry, which behaves far better when extrapolating past training lengths (context-window "stretching" tricks like YaRN operate directly on these angles).

The Systems View: KV Cache and FlashAttention

At inference you generate one token at a time. Recomputing K and V for the whole prefix at every step would be O(n²) per token; instead you cache them — the KV cache. Generation then costs O(n) attention per new token, but the cache grows linearly: for a 70B-class model at 128K context, it reaches tens of gigabytes — often more than the weights' own memory traffic budget. This is the real constraint behind context pricing, and why techniques like grouped-query attention (sharing K,V across heads) exist: they shrink the cache, not the math.

Training-side, the n×n score matrix doesn't fit in fast memory either. FlashAttention (2022) computes exact attention without ever materializing it — tiling the computation so scores live briefly in on-chip SRAM, using the online-softmax trick to accumulate correct results block by block. Not an approximation: a reordering of the same arithmetic around memory bandwidth. It's the reason today's 100K+ windows are practical at all.

The sub-quadratic graveyard deserves a mention: sparse attention, linear attention, sliding windows, and state-space models (Mamba) all trade the n² for something cheaper, and hybrids are increasingly common in production. But for frontier quality, full attention — made fast rather than approximate — still holds the crown.

What the Heads Actually Do

Attention is the most interpretable part of a Transformer, because the weights are right there to read. Mechanistic interpretability has mapped some of the machinery: induction heads — pairs of heads implementing "find the last time this pattern appeared and copy what came next" — emerge abruptly during training and appear to be a core mechanism of in-context learning, the model's ability to pick up a pattern from your prompt without any weight update. When you show three input→output examples and the model produces the fourth, you're watching attention heads run a learned algorithm over your examples. The next part shows where all of it comes from: training.

What's Next

We now have the full machine: tokens, geometry, and a stack of attention + feed-forward blocks that turns context into prediction. What we don't have is a reason for any of those trillions of numbers to be correct. Next time: training — next-token prediction as an objective, why compressing the internet forces a model to learn facts, grammar and a bit of reasoning, and what a "loss curve" actually measures.

---

Series roadmap: tokenization → embeddings → attention (you are here) → training → from autocomplete to assistant → inference. Questions? Tell me.