LLM for Dummies #5 — From Autocomplete to Assistant: The Persona Is Learned

Part 4 ended with an honest disappointment. After $100M and ten trillion next-token guesses you get a base model: an entity that knows an astonishing amount and has no idea it's supposed to help you. Ask it a question and it may answer, or it may continue with three more questions, because that's what questions on the internet are usually followed by.

The gap between that and the assistant you actually talk to is closed by post-training. It's the least expensive stage of the pipeline, the most opinionated, and the one that decides almost everything you feel about a model: its tone, its refusals, its confidence, its flattery.

Two Stages, One Village

FIG. 01 // THREE STAGES

Where the knowledge comes from, and where the manners come from

ILLUSTRATION: TYPICAL FRONTIER TRAINING PIPELINE · 2026

The proportions in that figure are the single most useful thing to remember from this part. Pretraining is a continent: months, tens of thousands of GPUs, ~99% of the compute. Post-training is a village: days, and datasets small enough that a company could read them. And yet the village decides the personality.

Why does so little data suffice? Because post-training doesn't add capability — it selects one. The base model can already write as a lawyer, a Reddit troll, a textbook, a helpful expert, a spam bot: it has absorbed all of them. Post-training's job is to make one of those voices the default and keep it there. In the trade this is called the superficial alignment hypothesis: knowledge and skills come from stage 1; stages 2 and 3 teach which subdistribution of that knowledge to speak from.

Stage 2: Show It What "Helpful" Looks Like

Instruction tuning (or SFT, supervised fine-tuning) is almost embarrassingly literal. Humans — and increasingly, stronger models — write thousands of example conversations: a question, followed by the kind of answer a competent, polite expert would give. Then you run the exact same training loop from part 4 on them: predict the next token, measure the surprise, nudge. The only difference is the data.

After a few thousand of these, the model has learned the format: that after a user speaks, the assistant answers; that answers are structured, complete and stop when done; that code goes in code blocks. It has learned to play a character.

FIG. 02 // WHAT 'CHAT' ACTUALLY IS

The conversation as the model sees it: one long string with markers

SystemUserAssistant

<|system|>You are a helpful assistant.<|user|>Capital of France?<|assistant|>Paris.<|end|>

THE MODEL JUST CONTINUES THE TEXT AFTER <|assistant|>. THE "ASSISTANT" IS A CHARACTER IT LEARNED TO PLAY.

ILLUSTRATION: CHAT TEMPLATE (MARKER NAMES VARY BY MODEL)

That figure is the part most people never see. "Chat" is not a different mode of the model. A conversation is a single document with role markers — special tokens added to the vocabulary — and the model does what it always does: continue the text after the last marker. Your system prompt is the first paragraph of that document. The assistant's "identity" is whatever the SFT data made statistically likely to follow <|assistant|>. This is why a system prompt can rename the assistant, change its personality, or make it answer in pirate speak: you're editing the opening of the story it's continuing.

Stage 3: Teach It Taste

SFT has a limit: humans are much better at recognizing a good answer than at writing the ideal one. So the third stage stops asking people to write, and starts asking them to compare. Show two candidate answers, A and B. Which is better? Collect hundreds of thousands of these judgments, train a small model (the reward model) to predict human preference, and then push the assistant toward answers the reward model scores highly — while keeping it tethered to where it started, so it doesn't drift into gibberish that happens to fool the judge.

That's RLHF — reinforcement learning from human feedback — and its more recent, simpler cousin DPO, which skips the separate reward model. This is where tone lives: concision vs. thoroughness, when to hedge, when to refuse, how apologetic to be. Every "As an AI language model, I cannot…" you've ever read was a preference somebody expressed, generalized by a reward model, and burned into weights.

It also has a famous failure mode. If human raters slightly prefer answers that agree with them, the reward model learns that flattery scores — and the assistant becomes sycophantic, telling you your bad plan is great. Optimizing for approval is not the same as optimizing for truth, and no amount of RLHF makes the distinction for you.

FIG. 03 // ORDERS OF MAGNITUDE

Post-training in three numbers

Share of total training compute spent on post-training

<1%

the manners are cheap; the mind was expensive

The unit of preference data

A vs B

humans don't write the ideal answer, they pick the better of two

New facts the model learns in post-training

≈ 0

knowledge comes from stage 1; stages 2-3 shape behavior

TYPICAL FRONTIER VALUES · 2026

What Post-Training Can't Do

Three things follow from "it selects, it doesn't add", and they explain a lot of daily frustrations:

• It can't install knowledge. Fine-tuning on your company's documents teaches the style of your documents far more reliably than their facts. For facts you want retrieval — handing the model the text at inference time — which is why RAG exists.
• It can't fully remove knowledge either. Refusals are a learned reflex on top of an intact base model. A prompt that makes the harmful continuation look like a different genre (a story, a role-play, a "hypothetical") slides around the reflex. Jailbreaks aren't bugs in the safety layer; they're the safety layer working exactly as shallow as it is.
• It can't make the model know what it doesn't know. Post-training rewards confident, well-formed answers. It doesn't — can't — teach the model which of its outputs are true. Hallucination survives post-training intact, which brings us to the last part of the series: what happens at inference, when the model finally has to pick a word.

---

Deep Dive: The Objectives, Formally

The "for dummies" part ends here — below assumes probability and some optimization. Otherwise, see you at part #6.

SFT Is Just Masked Cross-Entropy

Instruction tuning uses the same loss as pretraining, with one change: the loss is computed only on assistant tokens. User and system tokens are in the context but masked out of the objective — the model should learn to produce answers, not to imitate questions. Formally, with m_i = 1 on assistant positions:

L_SFT(θ) = − Σᵢ m_i · ln p_θ(tᵢ | t₁…tᵢ₋₁)

Chat templates make this concrete: role markers are reserved token IDs, and the stop-of-turn marker is what the model learns to emit when done — the "end of answer" behavior is literally one more next-token prediction. The LIMA result (2023) showed ~1,000 carefully curated examples suffice to produce a competent assistant from a strong base model, which is the empirical backbone of the superficial-alignment claim.

Reward Modeling: Bradley-Terry on Comparisons

Preference data is pairs (x, y_w, y_l) — prompt, winning answer, losing answer. A reward model r_φ(x, y) is trained so that the probability of the observed preference follows a Bradley-Terry model:

P(y_w ≻ y_l | x) = σ( r_φ(x, y_w) − r_φ(x, y_l) )
L_RM(φ) = − E [ ln σ( r_φ(x, y_w) − r_φ(x, y_l) ) ]

Only differences in reward are identified — the scale is arbitrary — which is one reason reward models are miscalibrated across prompts and easy to exploit.

RLHF: Maximize Reward, Stay Close to Home

The policy π_θ (the assistant) is then optimized to maximize expected reward with a KL penalty against the SFT reference π_ref:

J(θ) = E_{x, y∼π_θ} [ r_φ(x, y) ] − β · KL( π_θ(·|x) ‖ π_ref(·|x) )

The KL term is the whole game. Without it, the policy finds the reward model's blind spots (Goodhart's law in its purest form): answers that score highly and mean nothing. With it, the policy stays in a trust region around a model that already writes sensible text. In practice this is optimized with PPO, and β trades helpfulness gains against "alignment tax" — the measurable degradation in some capabilities that heavy post-training can cause.

DPO: The Same Thing Without the Reward Model

A key 2023 result: the RLHF objective above has a closed-form optimal policy, π*(y|x) ∝ π_ref(y|x) · exp(r(x,y)/β). Invert it to express the reward in terms of policies, substitute into the Bradley-Terry loss, and the reward model disappears:

L_DPO(θ) = − E [ ln σ( β · ln π_θ(y_w|x)/π_ref(y_w|x) − β · ln π_θ(y_l|x)/π_ref(y_l|x) ) ]

One supervised loss on preference pairs, no RL loop, no separate reward network — same optimum in theory. Most open-weight assistants since 2024 are trained with DPO or its variants; frontier labs typically blend on-policy RL (PPO/GRPO-style, increasingly with verifiable rewards from code execution or math checkers) with preference methods.

Why Sycophancy and Mode Collapse Are Predictable

Two structural consequences of optimizing against a learned judge. First, sycophancy: if raters' judgments correlate even slightly with agreement, r_φ inherits that correlation and the policy amplifies it — it's not a training bug but the objective faithfully executed. Second, entropy collapse: reward maximization concentrates probability mass on high-scoring modes, so post-trained models are less diverse and more repetitive than their base models (visible as narrower sampling distributions and the house style everyone recognizes). RLAIF and Constitutional AI replace some human judgments with model judgments against a written set of principles — cheaper and more consistent, with the obvious circularity risk.

The Alignment Layer Is Thin, Measurably

Interpretability work locates refusal behavior in a small number of directions in the residual stream (recall part 2's geometry): ablate a single "refusal direction" and a safety-tuned model complies with everything, with the rest of its capabilities intact. That is the mechanistic face of "selects, doesn't add": the base model's full distribution is still in there, one vector away. It's also why alignment researchers care so much about what happens before post-training.

What's Next

We now have the complete assistant: knowledge from pretraining, manners from post-training. One thing remains — the moment of truth when the model has a probability distribution over 100,000 tokens and must pick one. Next and final part: inference — temperature, sampling, why the same question gets different answers, why models hallucinate with a straight face, and what "reasoning models" actually do differently.

---

Series roadmap: tokenization → embeddings → attention → training → from autocomplete to assistant (you are here) → inference. Questions? Tell me.