LLM for Dummies #1 — Tokenization: Your AI Has Never Read a Word

This is the start of a new series. No market takes, no capex charts — just how large language models actually work, explained from the ground up, one piece at a time. The plan: tokenization, then embeddings, then attention, then training, then how a raw model becomes an assistant, then inference. By the end you'll be able to reason about what these systems can and can't do, instead of guessing.

We start where every LLM starts: with the uncomfortable fact that your AI has never read a word in its life.

The Model Has No Alphabet

When you send text to a language model, the model does not receive text. A component called the tokenizer sits in front of it, chops your sentence into chunks, and replaces each chunk with a number. The model receives the numbers. That's all it will ever see.

FIG. 01 // TEXT → NUMBERS

What the model actually receives

“Language models read tokens, not words.”

↓

Language14126␣models4211␣read1373␣tokens11460,11␣not539␣words4339.13

8 NUMBERS GO IN. THE MODEL NEVER SEES A SINGLE LETTER.

ILLUSTRATION: BPE TOKENIZATION · CL100K-STYLE VOCABULARY

Those chunks are called tokens. A token can be a whole word, a piece of a word, a punctuation mark, or even a single character. The mapping lives in a fixed vocabulary — a big lookup table decided once, before training, and frozen forever after. If English were the model's world, the vocabulary would be its periodic table: every text it will ever read or write is assembled from those ~100,000 pieces and nothing else.

Two details in that figure deserve a second look. First, the ␣ symbols: the space belongs to the token. "models" at the start of a sentence and "␣models" after another word are two different entries in the vocabulary. Second, the IDs are arbitrary. There's no logic connecting 4211 to "models" — it's just a row number in the table.

Why Not Just Use Letters? Or Words?

Both options exist, and both lose.

Letters would give you a tiny vocabulary (a few hundred symbols), but your sentence becomes brutally long — and, as we'll see later in the series, length is the enemy: the model's attention has to relate every position to every other position, and costs blow up fast.

Whole words keep sequences short, but the vocabulary explodes: every language, every name, every typo, every new word ever coined would need its own entry — and anything not in the table at training time becomes an unreadable hole.

Modern tokenizers use a compromise called Byte Pair Encoding (BPE). The idea is almost embarrassingly simple: start from single characters, then repeatedly merge the pair of chunks that appears most often in a huge pile of text, until you reach your target vocabulary size. Frequent things ("the", "tion", "models") end up as single tokens. Rare things ("antidisestablishmentarianism", your surname, Klingon) get spelled out from smaller pieces. Common text compresses; weird text costs more.

That one design choice — frequency decides the chunks — explains a surprising amount of LLM behavior.

The Language Tax

BPE learns its merges from training data, and training data is mostly English. Consequence: English compresses beautifully, other languages less so.

FIG. 02 // THE LANGUAGE TAX

Same sentence, different token bill

ILLUSTRATION: BPE TOKENIZATION · 2026

The same question costs about 60% more tokens in Italian than in English. Since everything is billed and bounded in tokens — API prices, context windows, generation speed — that gap is real money and real capacity. A "200K-token context window" holds roughly 150,000 English words, but noticeably fewer Italian ones, and dramatically fewer in languages like Thai or Amharic that the tokenizer barely learned. If you build products in a non-English language, you're paying an invisible tax decided years ago by a frequency counter.

Why the Model Can't Count the R's in "Strawberry"

Here's the classic party trick: ask an LLM how many "r"s are in strawberry and watch it stumble. People read this as "the AI is dumb." The real answer is more interesting: the model never saw the letters. The tokenizer handed it something like str + aw + berry — two or three opaque IDs. Asking it to count letters is like asking you to count the brushstrokes in a photo of a painting: the information was destroyed before you got it.

The same mechanism explains other "mysteries":

• Arithmetic feels alien. "12345" might tokenize as 123 + 45, so the model isn't manipulating digits — it's manipulating chunks that happen to contain digits. (Modern tokenizers special-case numbers into fixed groups precisely to soften this.)
• Rhymes and wordplay are hard. Sound and spelling live below the token level, where the model can't directly look.
• Glitch tokens exist. Vocabularies occasionally contain junk entries learned from weird corners of the internet (the infamous SolidGoldMagikarp), which sat nearly untrained and made models produce nonsense when triggered.

None of this is a reasoning failure. It's a sensory limitation, decided before the model learned anything at all.

The Numbers That Matter

FIG. 03 // ORDERS OF MAGNITUDE

Tokenization in three numbers

Entries in a typical modern vocabulary

~100K

every text ever processed maps to these chunks

English text per token, on average

~4 chars

≈ ¾ of a word; the yardstick behind every context window

Extra tokens for Italian vs English

+30-60%

same content, bigger bill — varies by tokenizer

TYPICAL VALUES: BPE TOKENIZERS (GPT/CLAUDE/LLAMA FAMILIES) · 2026

Keep these three in your head and a lot of LLM marketing gets easier to parse. When a pricing page says "$3 per million input tokens," you can now translate: about 750,000 English words, or one long novel, for three dollars — and less than that if you write in Italian.

---

Deep Dive: The Math Behind the Chopper

The "for dummies" part ends here — what follows assumes you're comfortable with university-level math notation. If that's not your thing, skip to the end and see you at part #2: you already have everything you need.

BPE, Formally

Start from a base vocabulary V₀ of all 256 byte values, so any UTF-8 string is representable by construction (no out-of-vocabulary symbol can ever exist). Given a training corpus C, BPE runs a greedy loop:

V ← V₀
repeat k times:
    (a, b) ← argmax over adjacent symbol pairs of count_C(a, b)
    V ← V ∪ { ab }          # new merged symbol
    replace every "a b" in C with "ab"

After k merges, |V| = 256 + k (typical k ≈ 50,000-200,000). The merge list is the tokenizer: at inference, you re-apply the same merges in the same order. Note what this objective is: each merge maximally reduces the corpus token count — BPE is a greedy dictionary-based compressor, a close cousin of LZ78. "Token count" and "compressed length" are the same quantity, which is why frequent strings cost 1 token and rare ones get spelled out.

One practical wrinkle: raw BPE would happily merge across spaces and punctuation, so real tokenizers first split text with a pre-tokenization regex (the GPT-4-style pattern splits on letter/digit/punctuation boundaries and caps digit runs at three — that's why "12345" becomes 123+45, and why number arithmetic behaves better than it did in GPT-2's era of arbitrary digit chunks).

The Vocabulary-Size Trade-off

Why ~100K entries and not 1K or 10M? Two costs pull in opposite directions.

Sequence length: a Transformer's self-attention costs O(n² · d) per layer for a sequence of n tokens with model width d. A smaller vocabulary → longer sequences for the same text → quadratic pain. If Italian takes 1.5× the tokens of English, its attention cost is ~2.25× for the same content.

Vocabulary size: the embedding matrix has |V| · d parameters, and the output softmax computes a distribution over all |V| entries at every generated position. At |V| = 10M and d = 12,288, the embedding table alone would be ~123B parameters — bigger than most entire models.

The optimum sits where the marginal compression gain of one more vocabulary entry stops paying for its parameter and softmax cost. Empirically that lands in the 10⁵ range for current model scales, and it drifts upward as models grow (frontier tokenizers moved from 50K to 100K-200K over four years).

Tokens as Compression: The Bits-Per-Byte View

Here's the cleanest way to see why tokenizer choice contaminates every benchmark. A language model assigns your text a probability; the standard quality metric, perplexity, is defined per token:

PPL = exp( −(1/N) Σᵢ ln p(tᵢ | t₁…tᵢ₋₁) )

But N is the token count — a tokenizer-dependent quantity. Two models with different tokenizers report incomparable perplexities on identical text. The fix is to normalize by something tokenizer-independent, the raw byte count B:

bits per byte = (N / B) · log₂ PPL

Total information is conserved: a tokenizer that compresses better (lower N/B) shifts work from "many easy predictions" to "fewer harder ones," but the product — bits needed to encode the text — is the fair, comparable number. This is also the honest lens on the language tax: Italian's higher N/B doesn't make models worse at Italian per se, but it does mean each context window holds fewer bytes of Italian meaning.

The Other Chopper: Unigram LM

BPE is not the only game in town. SentencePiece's Unigram tokenizer (used by Llama-family and T5 models) inverts the logic: instead of growing a vocabulary by merges, it starts from a large candidate set and models a segmentation s = (t₁ … tₙ) of the text probabilistically:

P(s) = Πᵢ p(tᵢ)

Tokenization of a string is the maximum-likelihood segmentation — found with Viterbi dynamic programming over all ways to split the string, O(L²) in string length with a max-token-length cap. Training alternates EM (re-estimate the p(tᵢ) given current segmentations) with pruning (drop the tokens whose removal least decreases corpus likelihood) until the target vocabulary size is reached. Unigram tends to produce slightly more linguistically natural splits and enables subword regularization — sampling suboptimal segmentations during training as data augmentation.

Why Glitch Tokens Happen, Mechanically

Close the loop on SolidGoldMagikarp: the tokenizer's training corpus (where merges are learned) and the model's training corpus are not the same. A string frequent in the first but nearly absent from the second gets a dedicated token whose embedding row receives almost no gradient updates during training. It stays close to its random initialization — a vector sitting off the learned manifold. Feed it to the model and every layer downstream processes garbage-in. The lesson generalizes: a token's competence is bounded by its training exposure, which is invisible from the outside.

What's Next

The tokenizer's numbers are just IDs — row 4211 says nothing about what "models" means. The magic starts in the next step, where each token ID gets turned into a vector of coordinates that encode meaning as geometry: words as points in space, where "king" and "queen" end up neighbors — and where the Viterbi segmentations and embedding rows we just met become the actual objects the model computes with. That's embeddings — part #2 of this series.

---

Series roadmap: tokenization (you are here) → embeddings → attention & the Transformer → training → from autocomplete to assistant → inference. Questions you'd like answered along the way? Tell me.