Educatifu
Open menu

Turning words into vectors

Networks only do arithmetic, so language must become numbers first. Embeddings turn words into vectors where direction encodes meaning.

CoreArtificial Intelligence~13 min

Before this lesson

  • Lesson: Backpropagation

A neural network only multiplies and adds numbers. So before a model can read a sentence, the sentence must become numbers. Doing that well is one of the quiet breakthroughs behind modern language AI.

Tokens: chunks of text

First, text is split into tokens — not always whole words. Common words are single tokens; rarer words break into pieces ("unbelievable" → un + believ

  • able). Tokenisation lets a fixed-size vocabulary cover any input, including words the model has never seen, by spelling them out of familiar parts.

Embeddings: tokens as vectors

Each token is then mapped to a vector of numbers — its embedding. Crucially, these vectors are not assigned by hand and they are not random. They are learned (by exactly the training loop from the last lesson), and learning pushes them into a remarkable arrangement: tokens with similar meanings end up pointing in similar directions.

That turns meaning into geometry. "king" and "queen" land near each other; "apple" and "orange" cluster elsewhere; "server" and "network" somewhere else again. Whole regions of the space come to mean royalty, or fruit, or computing. The model never stores a dictionary — meaning lives in the layout.

Measuring "related": cosine similarity

Because meaning is encoded as direction, the natural way to compare two embeddings is the angle between them, not the distance. Vectors pointing the same way are similar even if one is longer; vectors at right angles are unrelated. The cosine similarity captures exactly this — it is the dot product divided by the two lengths, giving 1 for identical direction, 0 for perpendicular, −1 for opposite. It is the workhorse behind search, recommendations, and "find me something like this".

Feel it

Below, a handful of words sit as vectors in a 2D toy space (real embeddings use hundreds of dimensions). Move the query vector and watch the ranking by cosine similarity: aim it at a cluster and every word in that cluster scores high at once. That is retrieval by meaning, in miniature.

Embeddings give the model static meaning — what each token means on its own. But "it" means different things in different sentences, and "bank" by a river is not a bank with an ATM. To resolve meaning in context, models need one more idea: attention.

Try it: words as vectors

queenkingprincebananalaptopappleserverorangenetwork

Cosine similarity to your query

  • queen1.00
  • king1.00
  • prince0.97
  • banana0.27
  • laptop0.15
  • apple0.07
  • server-0.04
  • orange-0.15
  • network-0.27

Cosine similarity measures the angle between vectors, not their length. Point your query toward a cluster and every word in it scores high at once — which is exactly how models retrieve “related” meaning. Real embeddings do this in hundreds of dimensions.

Key takeaways

  • A token is a chunk of text; an embedding is the vector a model uses to represent it.
  • Embeddings place similar meanings in similar directions, so geometry becomes a proxy for meaning.
  • Cosine similarity — the angle between vectors — is the standard way to measure how related two pieces of text are.

← All Artificial Intelligence lessons

Bring us the problem, not a perfect specification

Tell us what needs to change, who it affects and any important deadline. We will review the context and reply with useful next questions.

  1. 01Share contextDescribe the workflow, constraint or risk.
  2. 02Clarify togetherWe identify missing facts and useful options.
  3. 03Choose a startAgree a focused assessment or delivery step.
Start a conversation