Embeddings

Why This Matters

Embeddings are the quiet workhorse of applied AI. Semantic search, recommendations, deduplication, clustering, classification, and the entire retrieval half of RAG (AI-016) are embeddings underneath. If LLMs are the celebrity, embeddings are the plumbing every real system runs on. And they're cheap, fast, and available to any developer today.

Everyday Analogy

A map of meaning. On a city map, nearby dots are physically close. An embedding space is a map where nearby dots are close in meaning: "puppy" sits beside "dog," a stone's throw from "kitten," across town from "carburetor." Every new sentence gets GPS coordinates on this map. Once meaning has coordinates, "find similar things" becomes "find nearby dots," a problem computers solve brilliantly.

From Text to Numbers: A Tiny Worked Example

Real embeddings have hundreds to thousands of dimensions; here is a cartoon with 3, scoring each word on [animal-ness, size, domesticity]:

Distances now mean something: dog↔puppy are nearly touching; dog↔wolf differ mainly on one axis (domesticity); dog↔car are far apart. Real models learn ~1,536 such axes automatically from data. They're unlabeled and uninterpretable individually, but collectively capture tone, topic, syntax, and subtle shades no human would think to enumerate. Similarity is computed with a dot product or cosine: one multiplication per axis, summed, giving measurable meaning.

The Famous Party Trick — and What It Reveals

In good embedding spaces, directions carry meaning: the vector from "man" to "woman" is roughly parallel to the one from "king" to "queen," so king − man + woman ≈ queen. Paris − France + Japan ≈ Tokyo. Meaning has geometry: relationships are arrows, categories are neighborhoods. This is not a gimmick. It's the property that makes everything downstream work, and it's also where bias hides: directions learned from human text carry human stereotypes along for the ride (AI-062).

Where Embeddings Run Your Life

• Semantic search. Query and documents both become vectors; results are nearest neighbors. Finds "reset my password" when you typed "can't log in" — no shared keywords needed.
• Recommendations. "Users who liked X" = "vectors near X." Spotify, Netflix, and e-commerce run on neighborhood lookups.
• RAG retrieval (AI-016). The "R" is an embedding search over your knowledge base.
• Deduplication & clustering. Near-identical support tickets, duplicate listings, topic groupings — all distance thresholds.
• Classification on a budget. Embed, then run a tiny classifier on top — often 95% of the quality at 1% of the cost of an LLM call.

Practical Choices

Getting embeddings is one API call (OpenAI, Cohere, Voyage) or a free local model (sentence-transformers). The decisions that matter:

1. Model choice — leaderboards (MTEB) rank retrieval quality; domain-specific models exist for code, law, medicine.
2. What to embed — whole documents blur; embed chunks (paragraphs) for retrieval precision (chunking strategy returns in AI-016).
3. Dimension trade-off — more dimensions = finer meaning but more storage and slower search (AI-015 picks this up).
4. Same model both sides — queries and documents must share one embedding space; mixing models breaks similarity entirely.

Real-World Showcase

• Google Search has used embedding-based understanding (BERT and successors) since 2019, and billions of queries a day ride on meaning-vectors.
• Spotify's Discover Weekly embeds songs and listeners into a shared space; your Monday playlist is a neighborhood tour.
• Notion, Slack, and GitHub all shipped semantic search by embedding your workspace: the feature that finds the doc you described rather than the words you remembered.

Try It Yourself

1. Rank by meaning, yourself: order these by similarity to "The chef cooked a wonderful meal" — (a) "A delicious dinner was prepared," (b) "The chef was arrested," (c) "Rain is expected today." Your instant ranking (a, b, c) is what cosine similarity computes — you are an embedding model.
2. Trip up keyword search: search a shopping site for "footwear for running" vs "running shoes" and note any difference; semantic search would treat them identically.
3. Guess the axes: for songs instead of words, what might learned dimensions capture? (Tempo, mood, era, acousticness — Spotify's embeddings learn exactly such axes without being told.)

Common Mistakes

• Treating embeddings as keyword search. They match meaning; exact part-numbers and names sometimes need hybrid keyword+vector search (AI-015).
• Comparing vectors from different models — different maps, meaningless distances.
• Embedding whole books as one vector — meaning blurs; chunk first.
• Reading individual dimensions — axes are learned and distributed; only distances and directions are meaningful.
• Forgetting bias — embedding geometry inherits the prejudices of training text (AI-062).

Key Takeaways

• Embedding = content → coordinates on a learned map of meaning.
• Similar meaning = nearby vectors; similarity is a cheap arithmetic operation.
• Directions encode relationships (king − man + woman ≈ queen), and encode biases too.
• Search, recommendations, clustering, and RAG retrieval are all nearest-neighbor lookups.
• Chunk your content, pick one model for both sides, and you're production-ready.