Educatifu
Open menu

Training that generalises

A model that aces its training data can still fail on everything new. The real goal is generalisation — and the tension between underfitting and overfitting is the heart of it.

AdvancedArtificial Intelligence~14 min

Before this lesson

  • Lesson: Backpropagation

So far we have trained a model to reduce its loss on examples it has seen. But that is not actually what we want. A model that memorises its training set perfectly and fails on everything new is useless. What we care about is generalisation: doing well on data the model has never seen.

Measure what you care about

Because training error is not the goal, we do not trust it. We split the data: train on one part, and keep a validation set aside to estimate performance on unseen inputs. Validation error is the number that actually matters.

The two ways to fail

  • Underfitting — the model is too simple to capture the pattern. Both training and validation error stay high. A straight line trying to follow a curve.
  • Overfitting — the model is so flexible it fits the noise in the training data, not just the signal. Training error plummets toward zero while validation error climbs. The model has memorised, not learned.

The trade-off is unavoidable: as you add capacity (more parameters, more layers, more training), training error keeps falling — but validation error falls, bottoms out, then rises. That U-shape is the bias–variance trade-off, and the sweet spot is its lowest point.

See it

Below, fit a polynomial to fixed noisy data and drag the complexity up. Watch a low degree miss the trend, a moderate degree track it cleanly, and a high degree snake through every training point while its validation error blows up. The training-error bar only ever shrinks; the validation-error bar tells the truth.

Fighting overfitting

Practitioners have a toolbox for pushing that U-shape down and to the right so big models can still generalise:

  • More data — the single most reliable cure; noise averages out.
  • Regularisation — penalise large weights (weight decay) so the model prefers simpler explanations.
  • Dropout — randomly switch off neurons during training so no single path can memorise.
  • Early stopping — halt when validation error starts rising.

These are why models with billions of parameters — far more than their training examples in some views — can still generalise. With generalisation understood, we can look at how the largest models are actually trained: the pretraining objective.

Try it: underfit vs overfit

training validation (unseen)
training error0.018
validation error0.003

Good fit — low error on the unseen validation data too.

Training error always falls as complexity rises — but validation error, on data the model never saw, is U-shaped. The best model minimises that, not training error.

Key takeaways

  • The goal is low error on unseen data, not on the training data — so we hold out a validation set to measure it.
  • Underfitting means the model is too simple; overfitting means it memorised the training data and won't generalise.
  • More capacity always lowers training error but validation error is U-shaped — the best model sits at the bottom of the U.

← All Artificial Intelligence lessons

Bring us the problem, not a perfect specification

Tell us what needs to change, who it affects and any important deadline. We will review the context and reply with useful next questions.

  1. 01Share contextDescribe the workflow, constraint or risk.
  2. 02Clarify togetherWe identify missing facts and useful options.
  3. 03Choose a startAgree a focused assessment or delivery step.
Start a conversation