AG News: news headlines in 4 topics (World, Sports, Business, Sci/Tech). We take 2,000.
| Representation | Shape | Zeros |
|---|---|---|
| One-hot BoW | 2,000 × 11,428 | 99.73% |
| BoW counts | 2,000 × 11,428 | 99.73% |
| TF-IDF (no stop words) | 2,000 × 11,176 | 99.81% |
Each headline uses ~20 of 11,000 possible words.
That's why we call it sparse.
How close are two vectors? Look at the angle between the arrows.
| S1–S2 (cat / dog) | S1–S3 (cat / stocks) | |
|---|---|---|
| BoW counts | 0.75 | 0.40 |
| TF-IDF | 0.54 | 0.21 |
TF-IDF pulls apart sentences that only share "the" and "sat".
But: cat and dog are separate columns. To a sparse vector they are as different as cat and stocks.
Sparse vectors know which words appear, not what they mean.
A bottle of tezgüino is on the table.
Everybody likes tezgüino.
Tezgüino makes you drunk.
We make tezgüino out of corn.
You've never seen the word, but you know it's an alcoholic drink.
Distributional hypothesis: words in similar contexts have similar meanings.
Train a model on billions of words to predict a word from its neighbours:
the ___ chased the mouse → cat
The model learns one vector per word as a side effect, and that vector is the embedding.
Same family: one vector per word, learnt from context.
| One-hot word | GloVe | |
|---|---|---|
| Length | 400,000 | 100 |
| Non-zero entries | 1 | 100 |
| cos(cat, dog) | 0 | 0.88 |
| cos(cat, mat) | 0 | 0.16 |
| cos(stocks, market) | 0 | 0.77 |
Rhyming doesn't matter; meaning does.
| Analogy | GloVe's answer |
|---|---|
| king − man + woman | queen |
| paris − france + spain | madrid |
| walking − walk + swim | swimming |
| bigger − big + small | larger (then smaller) |
Directions in the space encode relations: gender, capital of, -ing form.
Not perfect, and it learns biases from its training text too.

100 dimensions squeezed into 2 (PCA). Countries sit next to their capitals; numbers huddle together; verbs are spread out.
Averaging word vectors is crude. Instead, a model reads the whole text and outputs one vector.
all-MiniLM-L6-v2 (sentence-transformers, 2019+)
It's a small Transformer, and how it works inside is next session.

Same headlines, same colours, both squeezed to 2D with UMAP. TF-IDF: one blob with fuzzy borders. MiniLM: separate islands. (Nearest-neighbour topic accuracy: 78% vs 86%.)
Nearest MiniLM neighbours that share no content words. TF-IDF scores them 0.
| Headline | Closest neighbour | cos |
|---|---|---|
| Transactions BASEBALL Baltimore (AL): Signed OF B.J. Surhoff… | Hudson Traded to Braves ATLANTA… | 0.55 |
| Hurricane losses may reach $10bn… | Hurricanes Leave Huge Economic Footprint… | 0.52 |
| Deutsche Tele offers 3 bn to buy back T-Online… | Telefonica rejects 2 bln euro bid… | 0.51 |
| Just 3 of 18 candidates for president attend debate KABUL… | Afghan Poll Is Mostly Calm… | 0.51 |
For a bag of words, Hurricane ≠ Hurricanes and Kabul has nothing to do with Afghan.
Query "oil prices" →
| Similarity | Headline |
|---|---|
| 0.72 | OPEC President Says Politics Pushing Oil Prices Higher… |
| 0.70 | Oil Holds at $47 as Iraq Resumes Exports… |
| 0.68 | Oil Falls $3, Winter Fuel Inventories Up… |
Comparing against every vector is fine for 2,000 documents; not for 100 million.
Approximate nearest neighbours: trade a little accuracy for huge speed.
HNSW: a layered graph. Big jumps on top, then refine downwards.
Vector databases store vectors + metadata and index them: FAISS, pgvector, Qdrant, Chroma, Pinecone…
RAG (retrieval-augmented generation) = semantic search that feeds an LLM.
| bites | dog | man | |
|---|---|---|---|
| dog bites man | 1 | 1 | 1 |
| man bites dog | 1 | 1 | 1 |
Identical vectors, very different news.
Sentences are sequences: order and length vary, and both matter.
GloVe's nearest neighbours of bank:
banks, banking, credit, investment, financial, securities, lending, funds, ubs, finance
The river bank sense has vanished: the vector is dominated by the most frequent meaning.
The vector for the token bank in four sentences:
GloVe (static)
| river 1 | river 2 | money 1 | money 2 | |
|---|---|---|---|---|
| river 1 | 1 | 1 | 1 | 1 |
| money 1 | 1 | 1 | 1 | 1 |
MiniLM (contextual)
| river 1 | river 2 | money 1 | money 2 | |
|---|---|---|---|---|
| river 1 | 1 | 0.82 | 0.72 | 0.65 |
| money 1 | 0.72 | 0.75 | 1 | 0.82 |
MiniLM gives bank a different vector depending on the words around it.
What if each word's vector depended on its neighbours in the sentence?
That's attention: the core of every Transformer and every LLM.

github.com/ber2/enode-talk
Notebook + code to run everything on your laptop
0:00. Introduce yourself and the series: this is session 1, attention is next.
1:00
2:30. Come back to this table at every section.
4:00
GPT-4o's tokenizer did ~200,000 merges.
6:00. Notebook section 1: type a sentence live, ask the audience for one.
11:00
14:00
20:00. Notebook section 2: tabs for the toy example, then the AG News sample.
25:00
25:00
30:00. Notebook section 3: ask the audience for words. Try football, madrid, pizza.
33:00
Notebook section 4: hover over points, find a Sci/Tech dot among the Business ones.
40:00
Notebook section 5: take queries from the audience. "football transfer" returns American football: the corpus is US news.
43:00
43:00
Notebook section 6.
48:00