A transformer is an architecture; it does nothing until it is trained. Large language models are trained by pretraining: reading enormous amounts of text and, at every position, trying to predict the next token given all the tokens before it.
Why next-token prediction is so powerful
It sounds trivial, even boring. It is neither. To predict the next token well across billions of passages, a model is quietly forced to learn:
- grammar — to get word order and agreement right;
- facts — "The capital of France is ___" only has one good continuation;
- reasoning — completing a chain of argument or a maths step;
- style and register — matching the tone of what came before.
None of these are taught directly. They are emergent — capabilities that appear because they help minimise the one loss. And because the correct next token is simply the next word already in the text, pretraining is self-supervised: the data labels itself. That is what lets it scale to trillions of tokens without an army of annotators.
The training loop, unchanged
Under the hood it is everything from the earlier lessons: embed the tokens, run them through the transformer's attention and feedforward layers, produce a probability distribution over the vocabulary for the next token, score it against the actual next token with a loss, and update every weight by backpropagation and gradient descent. The same engine — just at breathtaking scale.
Generating text: one token at a time
Once trained, the model generates by running that prediction forward. Given a prompt, it produces a probability for every possible next token, picks one, appends it, and repeats — each new token becoming context for the next.
How it "picks" matters. Temperature reshapes the probability distribution before sampling: low temperature concentrates on the single most likely token (focused, repetitive, near-deterministic); high temperature flattens the odds (surprising, creative, sometimes incoherent). This is why the same prompt can give different answers, and why a "temperature" dial appears in almost every model API.
Try it below: adjust the temperature to reshape the next-token odds for a short prompt, then press Sample to draw one.
A pretrained model is a remarkable next-token predictor — but it is not yet a helpful assistant. It will happily continue a prompt in any direction, including unhelpful or unsafe ones. Turning this raw capability into something that follows instructions and behaves well is the final step: alignment.