Educatifu
Open menu

From base model to assistant

A pretrained model knows language but isn't helpful. Instruction tuning and preference tuning (RLHF) turn a raw next-token predictor into an assistant.

ExpertArtificial Intelligence~15 min

Before this lesson

  • Lesson: Pretraining — predict the next token

The model that comes out of pretraining is called a base model. It is an extraordinary next-token predictor and a poor assistant. Ask it a question and it may continue with more questions, because in its training data questions are often followed by more questions. It has capability without direction. Alignment is the process of pointing that capability at being genuinely useful.

Step 1: instruction tuning

First the base model is fine-tuned — trained further, at a low learning rate — on a curated dataset of instruction → response pairs written or vetted by people: "Summarise this email." → (a good summary.) This is still next-token prediction, but now on examples of the behaviour we want. The model quickly learns the pattern "when given a request, produce a helpful answer" instead of merely continuing the text. After instruction tuning it feels like an assistant.

Step 2: preference tuning (RLHF)

Instruction tuning teaches a correct answer; it does not teach which of several plausible answers people actually prefer. That is what preference tuning adds, classically via RLHF — reinforcement learning from human feedback:

  1. The model generates several candidate responses to a prompt.
  2. People (or a model trained to imitate them) rank the responses.
  3. A reward model learns to predict those human preferences.
  4. The assistant is optimised to produce responses the reward model scores highly.

The result is a model tuned toward being helpful, honest and harmless — better at following the spirit of a request, declining unsafe ones, and matching the tone people want. Newer methods such as DPO (direct preference optimisation) reach a similar place more simply, optimising directly on preferred-vs-rejected pairs without a separate reinforcement-learning loop.

Why this matters

Alignment is where a model's values and manners come from — and where much of the hardest, most consequential work in modern AI lives. The same base model can become a careful assistant or a reckless one depending entirely on this stage. Pretraining gives a model what it can do; alignment shapes what it will do.

You now have the whole pipeline end to end. The final lesson steps back from how models are built to how to use them well — their real limits, and the practical levers that matter in production.

A pipeline from pretraining to instruction tuning to preference tuning (RLHF), producing an aligned assistant.
The stages that turn a base model into an assistant.

Key takeaways

  • A base model predicts plausible text; alignment makes it follow instructions and behave helpfully and safely.
  • Instruction tuning fine-tunes on curated request→response examples so the model learns to answer, not just continue.
  • Preference tuning (RLHF/DPO) optimises toward responses humans rate higher, shaping helpfulness, honesty and safety.

Sources

  1. [1]Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., 2022)arxiv.org
  2. [2]Direct Preference Optimization (Rafailov et al., 2023)arxiv.org

← All Artificial Intelligence lessons

Bring us the problem, not a perfect specification

Tell us what needs to change, who it affects and any important deadline. We will review the context and reply with useful next questions.

  1. 01Share contextDescribe the workflow, constraint or risk.
  2. 02Clarify togetherWe identify missing facts and useful options.
  3. 03Choose a startAgree a focused assessment or delivery step.
Start a conversation