RAG in Practice: Chunking, Hybrid Search & Reranking

Why This Matters

AI-016 named the failure points: retrieval misses, bad chunks, ignored context. This lesson is the practitioner's answer to the first two. Every team that ships "chat with your docs" hits the same wall a few weeks in: the naive pipeline (fixed-size chunks, pure vector top-k) works great in the demo and then quietly fails on the questions that matter, an exact part number, a rare acronym, a passage split mid-sentence. The fix is not a bigger model. It is a better retrieval pipeline, and the three techniques in this lesson, deliberate chunking, hybrid search, and reranking, are what separate teams that debug RAG quality systematically from teams that keep tweaking the prompt and wondering why nothing improves.

Everyday Analogy

Imagine a library that just got a delivery of encyclopedias, cut into random 200-word slices, and shelved without any care for where a slice starts or ends. The librarian who only follows a mental map of "what topic is this about" (pure vector similarity) will miss the shelf if you ask for the exact serial number printed on page 340 (that is where you need keyword search too). And even after grabbing the right ten slices off the shelf, you still need a sharp-eyed assistant to skim those ten and hand you the best three before you sit down to read (that is the reranker). Chunking is how carefully the encyclopedia was cut in the first place; hybrid search is having both a topic-map librarian and a keyword-index librarian working the request together; reranking is the final sharp-eyed pass before the material reaches you.

Chunking Strategies

AI-016 treats chunking as a single lever ("chunk on structure, overlap edges"). In production, chunking is actually a decision with several distinct strategies, each with real tradeoffs:

• Fixed-size chunking. Split every document into chunks of N tokens (commonly 200–500, per AI-015), with a fixed overlap (commonly 10–20% of chunk size) between consecutive chunks so a sentence split across a boundary still appears whole in at least one chunk. Simple, fast, and predictable, but blind to document structure: a fixed 300-token window can slice a table in half or separate a heading from the paragraph it introduces.
• Recursive character/token splitting. Used by tools like LangChain's RecursiveCharacterTextSplitter, this tries a hierarchy of separators (paragraph breaks, then sentence breaks, then word breaks) and only falls back to a harder cut when a natural boundary is not found within the size limit. It produces chunks that respect paragraph and sentence boundaries far more often than pure fixed-size splitting, at similar implementation cost.
• Semantic chunking. Instead of a fixed token count, this approach embeds sentences (or small groups of sentences) and looks for points where the embedding similarity between consecutive sentences drops sharply, splitting there. The result is chunks that align with actual topic shifts in the text rather than an arbitrary token count, at the cost of an extra embedding pass over the document during indexing.
• Structure-aware chunking. For documents with explicit structure (Markdown headings, HTML tags, PDF sections), splitting along that structure, one chunk per section or subsection, keeps titles attached to their content and respects the author's own organization, often the single highest-leverage chunking change for structured knowledge bases like wikis and API docs.

The size/overlap tradeoff: smaller chunks retrieve more precisely (less irrelevant text riding along with the answer) but lose surrounding context the LLM might need to interpret them correctly; larger chunks preserve context but dilute the embedding's specificity and burn more of the context window (AI-061) per retrieved passage. Overlap trades a small amount of duplicate content for protection against losing meaning at a chunk boundary. There is no universal correct number, teams tune chunk size and overlap against their own corpus and a held-out set of test questions, exactly the evaluation discipline AI-016 calls for.

Hybrid Search

Pure vector similarity search, the approach in AI-015's basic pipeline, matches on meaning, which is exactly why it misses exact identifiers: a part number like "TS-4102," a product SKU, or an uncommon proper noun does not have a distinctive enough embedding neighborhood to reliably surface in a top-k vector query. Traditional keyword search solves this precisely but misses paraphrases the other direction ("annual leave" versus "vacation days," AI-016's own example). Production retrieval runs both and merges the results:

• BM25 is the standard keyword-search ranking algorithm (a refinement of TF-IDF), scoring documents by term frequency, weighted so rare terms count more and adjusted for document length. It is what most production hybrid pipelines use as the keyword-search side, since it needs no training and runs fast at scale.
• Reciprocal Rank Fusion (RRF) is the standard way to merge two ranked lists (the BM25 ranking and the vector-similarity ranking) into one combined ranking, without needing the two systems' raw scores to be on comparable scales. Each document gets a score of 1/(k + rank) from each list it appears in (k is a small constant, commonly 60), and the scores are summed; a document that ranks highly in both lists rises to the top of the fused result, while a document that only one method found still gets a fair chance to surface.

Vector databases increasingly support this natively: Weaviate, Qdrant, and Elasticsearch/OpenSearch all offer built-in hybrid search combining a BM25-style keyword score with vector similarity, usually with RRF or a tunable weighted blend as the fusion method, so teams do not need to hand-roll the merge logic.

Reranking

Even a well-tuned hybrid search returns a ranked list from fast, approximate methods, vector similarity and BM25 are both designed to scan millions of documents quickly, which means both trade some precision for speed. A reranker adds a second, slower, far more accurate pass over a small candidate set:

1. Hybrid search returns the top 50–100 candidates cheaply.
2. A cross-encoder model, one that takes the query and a candidate passage together as a single input and directly scores their relevance, rather than comparing two separately-computed embeddings, reranks just those 50–100 candidates.
3. The top 3–10 reranked results, now much more reliably the actually most relevant passages, are what get assembled into the LLM's prompt (AI-016's step 3).

Cross-encoders are more accurate than embedding similarity because they let the model directly attend across the query and passage together, rather than compressing each into a fixed vector first and comparing after the fact, but they are too slow to run over an entire corpus, hence using them only on the short candidate list a fast retriever already narrowed down. Real reranker options teams use in production:

The Assembled Production Pipeline

Putting chunking, hybrid search, and reranking together, the pipeline this lesson adds on top of AI-016's basic shape looks like:

Offline: structure-aware or recursive chunking → embed each chunk → index in a vector database and a BM25 keyword index (often the same store, via Weaviate/Qdrant/OpenSearch's hybrid support).

Online, per question: embed the query → run vector search and BM25 search in parallel → fuse the two ranked lists with reciprocal rank fusion → take the top 50–100 fused candidates → rerank with a cross-encoder → take the top 3–10 → assemble the prompt exactly as AI-016 describes → generate.

This is strictly more expensive per query than naive top-k vector search (one extra keyword search, one reranking pass), but the latency cost is small (reranking 50-100 short passages typically adds tens to a couple hundred milliseconds) relative to the accuracy gained, which is why it is the default architecture in serious production RAG systems rather than an optional upgrade.

Real-World Showcase

• Cohere Rerank is used as a bolt-on reranking layer by teams running RAG on top of any vector database, Pinecone, Weaviate, or a custom store, specifically because it requires no infrastructure to self-host and measurably improves top-k precision in published benchmarks.
• Elasticsearch and OpenSearch's native hybrid query support (combining their long-standing BM25 engine with vector kNN search) is widely adopted by enterprises that already run Elastic for search and want to add RAG without standing up a separate vector database.
• LangChain's EnsembleRetriever and LlamaIndex's hybrid retrieval modules are the most common open-source implementations of the fuse-then-rerank pattern, used as the default retrieval layer in countless production RAG applications rather than teams hand-writing reciprocal rank fusion from scratch.

Try It Yourself

1. Take a document you know well and manually chunk it three ways: fixed-size (say 300 tokens, no regard for structure), recursive (respecting paragraph breaks), and structure-aware (one chunk per heading). Compare how a mid-document fact reads in each chunking's version, does it retain enough context to stand alone?
2. Design a hybrid query for a corpus you know: what's an example question where keyword search (an exact name, code, or number) would clearly beat vector search, and one where vector search (a paraphrased concept) would clearly beat keyword search? That pair of examples is the argument for why you need both.
3. Sketch a reranking budget: if your hybrid search returns 50 candidates and your reranker costs $0.001 per reranked passage, what's the added cost per query, and how does that compare to the cost of one wrong answer reaching a user?

Common Mistakes

• Using fixed-size chunking on structured documents (Markdown, API docs, contracts) when a structure-aware split would keep headings attached to their content for a similar amount of engineering effort.
• Skipping hybrid search entirely and relying on vector-only retrieval, then being surprised when exact product codes, names, or numbers never surface in results.
• Adding a reranker but feeding it too few candidates from the first-stage retriever, if hybrid search only returns the top 5, the reranker has nothing meaningfully better to select from.
• Treating chunk size as a one-time decision instead of a tunable parameter, teams that never revisit chunk size against a golden evaluation set (AI-016, AI-025) leave retrieval quality on the table.
• Assuming reranking fixes a bad first-stage retriever. A cross-encoder can only reorder the candidates it receives; if the right passage isn't in the top 50-100 from hybrid search, no reranker recovers it.

Key Takeaways

• Chunking strategy (fixed-size, recursive, semantic, or structure-aware) determines whether retrieved passages carry coherent, self-contained meaning; structure-aware splitting is often the highest-leverage upgrade for organized knowledge bases.
• Hybrid search merges BM25 keyword search with vector similarity via reciprocal rank fusion, catching both exact identifiers and paraphrased concepts that either method alone would miss.
• Rerankers (Cohere Rerank, bge-reranker, and similar cross-encoders) take a first-pass candidate list and reorder it with far higher precision before it reaches the LLM, at a modest latency cost.
• The production pipeline, chunk → hybrid search → fuse → rerank → generate, is strictly more than AI-016's basic retrieve-then-generate shape, and is what separates a working demo from a system that survives real queries.
• Debug by stage: a bad answer can stem from bad chunking, a retrieval miss, or a reranker never seeing the right candidate, each has a different fix.