Vector Databases

Why This Matters

AI-014 gave every document coordinates on a meaning-map. Now the engineering question: with 10 million documents, how do you find the 5 nearest neighbors fast? Comparing against every vector, one by one, takes seconds per query — dead on arrival for products. Vector databases solve exactly this, and they are the storage layer of RAG (AI-016), semantic search, and recommendations. When your chatbot "searches the company docs," a vector database answered.

Everyday Analogy

Finding the book most similar to yours in a library. A brute-force librarian compares your book to every one of a million volumes — accurate, and takes a week. A smart librarian first walks you to the right neighborhood ("fiction, French, 19th century") then compares against only the few hundred shelved nearby. You might miss one odd book shelved elsewhere (that's the "approximate"), but you get excellent matches in seconds. Vector databases are the smart librarian: they pre-organize the space so queries only visit promising neighborhoods.

The Core Trick: Approximate Nearest Neighbor (ANN)

Exact search over N vectors costs N comparisons per query. ANN indexes cut this to a tiny fraction by accepting near-perfect results:

• HNSW (the dominant index): builds a multi-layer "highway network" over the vectors: coarse express links on top, local streets below. A query rides the highways to the right region, then explores locally. Milliseconds over millions of vectors, ~95–99% recall.
• IVF: clusters vectors into buckets; search only the closest few buckets.
• Quantized variants: compress vectors (echoes of AI-065) to fit more in memory.

The universal dial is speed vs recall: visiting more neighborhoods finds truer neighbors but takes longer. Production teams tune this dial with measurements, not guesses (AI-025's eval mindset).

The Standard Pipeline

Every vector-search product runs these four steps:

1. Chunk. Split documents into passages (~200–500 tokens each, AI-014's advice), small enough to be precise, big enough to carry meaning.
2. Embed. Run every chunk through one embedding model; store vector + the original text + metadata (source, date, author).
3. Index. The database builds its HNSW/IVF structure over the vectors.
4. Query. Embed the user's question with the same model, ask for top-k nearest chunks, get back the passages in milliseconds.

This pipeline is the "R" of RAG — the next lesson plugs an LLM onto its output.

Choosing Your Vector Store

The honest industry advice: start with pgvector or Chroma; adopt a dedicated engine when scale or ops genuinely demand it. Many production RAG systems never outgrow Postgres.

Two Features That Separate Toys from Products

• Metadata filtering. "Nearest neighbors where team = legal and year ≥ 2024." Real queries almost always combine similarity with filters, and databases differ sharply in how well they filter without wrecking recall.
• Hybrid search. Vectors miss exact identifiers ("error TS-4102", part numbers, names); keyword search misses paraphrases. Production search runs both and merges results (often with a reranker), the fix for AI-014's "part-number problem."

Real-World Showcase

• Notion AI searches your workspace semantically — every page you write is chunked, embedded, and indexed within moments.
• Shopify runs vector search across billions of product embeddings for semantic shopping queries.
• pgvector's rise is the quiet story: a free Postgres extension became the default choice for startups, exactly because "the boring option that's already in your stack" usually wins.

Try It Yourself

1. Be the ANN index: think of a friend, then find "similar people" by first picking a neighborhood (same job? same hobby?) and only comparing within it. Notice you might miss a similar person from another circle — you just traded recall for speed.
2. Design the pipeline on paper for your own use case (your notes, your company wiki): what's a chunk? what metadata would you store? what would a hybrid query look like? These three answers are 80% of a real design doc.
3. Spot the hybrid need: search your email for a person's name (keyword wins) vs "that message about rescheduling the offsite" (semantic wins). Every real corpus needs both — now you know why.

Common Mistakes

• Reaching for a dedicated vector DB on day one. pgvector or Chroma serves most projects; add infrastructure when metrics demand it.
• Ignoring the recall dial. Default index settings silently trade accuracy; measure retrieval quality on your own data (AI-025).
• Skipping metadata design — bolting on filters after indexing millions of chunks is painful; plan them first.
• Pure vector search for everything — exact codes and names need hybrid keyword+vector.
• Forgetting re-indexing — documents change; stale vectors serve stale answers (freshness, AI-006).

Key Takeaways

• Vector DBs answer "top-k most similar vectors" at scale: the productionization of embeddings.
• ANN indexes (HNSW) trade a sliver of recall for 1000× speed; the dial is yours to tune.
• Pipeline: chunk → embed → index → query, same embedding model on both sides.
• Start boring (pgvector/Chroma); graduate to dedicated engines with scale.
• Metadata filters + hybrid search separate production systems from demos.