Attention and transformers

Attention lets each word gather meaning from the words around it. Stack it and you get the transformer — the architecture behind every modern LLM.

CoreArtificial Intelligence~15 min

Before this lesson

Lesson: Turning words into vectors

Embeddings give each token a fixed meaning, but language is contextual. In "the cat sat because it was tired", the word "it" only makes sense once you know it refers to the cat. The mechanism that lets a model resolve this — and that turned out to scale astonishingly well — is attention.

Every word looks at every word

Attention lets each token gather information from all the other tokens in the sequence, keeping what is relevant and ignoring the rest. For the token "it", the model learns to look at "cat" and copy in its meaning. It does this for every token simultaneously, in parallel.

How a weight is computed

For each token, the model derives three vectors from its embedding: a query (what am I looking for?), a key (what do I offer?), and a value (what I'll hand over if chosen). To decide how much token A should attend to token B:

score(A, B) = (queryA · keyB) / √d
weights     = softmax(scores over all tokens)   # sum to 1
output(A)   = Σ weights · valueB

The dot product measures how well A's query matches B's key; dividing by √d keeps the numbers stable; softmax turns the raw scores into a clean distribution that sums to 100%. Each token's new representation is the weighted blend of everyone's values — context, folded in.

See attention resolve a pronoun

Below, click any word to make it the query and watch where its attention goes. By default "it" is the query, and its attention concentrates on "cat" — the model has resolved the reference. (The token vectors here are hand-set toy features so the maths is visible; a real model learns them.)

From attention to transformers

A transformer is what you get when you stack this: many attention layers, and within each layer several attention "heads" that look for different kinds of relationship at once (one head might track grammar, another long-range topic). Between attention layers sit ordinary feedforward neurons — the same weighted sums and activations from the Foundations track. Add embeddings at the bottom and a prediction head at the top, train the whole thing by backpropagation on oceans of text, and you have a large language model.

That is the arc of this track in one sentence: a neuron, stacked into networks, trained by gradient descent and backpropagation, fed language through embeddings, and given context by attention. Everything else in modern AI is scale, data, and refinement on top of these foundations.

Try it: attention

Click a word to make it the query

“it” attends most to “cat” (27%).

cat27%
tired11%
sat10%
The8%
because8%
was8%

Each word carries a vector. The query’s vector is compared to every other word’s by dot product, scaled and softmaxed into weights that sum to 100%. Here the pronoun “it” lines up with “cat”, so attention pulls the cat’s meaning into “it”. Stack this across many layers and heads and you have a transformer.

Key takeaways

Attention lets every token look at every other token and pull in the context it needs.
A weight for each pair is computed as a scaled dot product of query and key vectors, softmaxed to sum to 1.
A transformer stacks many attention layers and heads; this architecture is what powers modern large language models.

Sources

[1]Attention Is All You Need (Vaswani et al., 2017)arxiv.orgThe paper that introduced the Transformer.
[2]The Illustrated Transformer — Jay Alammarjalammar.github.io

← All Artificial Intelligence lessons

Bring us the problem, not a perfect specification

Tell us what needs to change, who it affects and any important deadline. We will review the context and reply with useful next questions.

01Share contextDescribe the workflow, constraint or risk.
02Clarify togetherWe identify missing facts and useful options.
03Choose a startAgree a focused assessment or delivery step.

Start a conversation