Where RAG stops being RAG and starts being a search problem

The RAG ceiling

Most RAG systems hit a quality ceiling around deployment number three. The first two work well enough that everyone is excited. The third one, usually with messier data or more nuanced queries, starts returning irrelevant chunks and the team starts tuning chunk sizes and top-k values and wondering why it isn't working.

The ceiling is not a RAG problem. It's a search problem that teams are trying to solve with RAG tooling.

What RAG actually is

RAG — retrieval-augmented generation — is a pattern: retrieve relevant context, stuff it into a prompt, generate a response. The retrieval step is almost always described as "embedding-based similarity search." But embedding similarity is just one retrieval mechanism. It's the right one for some queries and the wrong one for many others.

After building production RAG systems across healthcare, legal, e-commerce, and internal knowledge bases, we've stopped using RAG as a system descriptor. We call it what it is: a search and retrieval stack. The question is always "what retrieval mechanism does this query type need," not "which vector similarity threshold should we tweak."

The four retrieval mechanisms

Dense retrieval — what most people mean by RAG. Embed the query and the corpus, retrieve nearest neighbours by cosine similarity. Works well for conceptual similarity ("documents about contract renewal") and poorly for exact match ("the clause on page 12 of contract #4872").

Sparse retrieval — BM25 and its variants. Old-school keyword search. Works extremely well for exact terminology, product codes, proper nouns, and technical jargon. Frequently outperforms dense retrieval for enterprise knowledge bases where users search with precise vocabulary.

Structured retrieval — SQL or filter-based lookups against document metadata. When the query has a structured component ("invoices from Q3 2024 over $50,000"), no amount of embedding tuning will match the precision of a parameterised query against an indexed metadata store.

Graph traversal — for knowledge that lives in relationships rather than documents. "What does our insurance policy say about pre-existing conditions when the patient also has a referral from a primary care physician?" is a multi-hop traversal problem, not a similarity search problem.

The hybrid architecture

Production systems almost always need at least two retrieval mechanisms working together. The pattern we've converged on:

1. Query classification — route the query to the right retrieval path based on its type. A lightweight classifier (often a simple prompt) determines whether the query needs dense, sparse, structured, or hybrid retrieval.

2. Parallel retrieval — run the relevant mechanisms in parallel and collect candidate results from each.

3. Re-ranking — use a cross-encoder re-ranker (Cohere Rerank, bge-reranker, or a custom model) to score all candidates together. Re-ranking consistently improves precision at the cost of ~50ms latency, which is almost always worth it.

4. Citation attribution — track which source chunk contributed to which part of the answer. This is table stakes for enterprise deployments and requires metadata threading from retrieval through generation.

The chunk size fallacy

The most common RAG tuning activity is adjusting chunk size. Teams try 256 tokens, then 512, then 1024, then back to 512 with 20% overlap. This is usually the wrong thing to optimise.

Chunk size matters, but it's downstream of document structure. A legal contract chunked at 512 tokens by token count will frequently split clauses mid-sentence. The right chunking unit is the semantic unit of the document — a section, a clause, a paragraph — not a fixed token count.

Document-aware chunking — splitting on structural boundaries like headings, paragraph breaks, or XML elements — consistently outperforms fixed-size chunking on retrieval precision, usually by 15–30 percentage points on a well-designed eval set.

Evaluating retrieval separately from generation

The most important architectural discipline in a search and retrieval stack is evaluating retrieval and generation as separate systems. Most teams measure end-to-end: "was the final answer correct?" This conflates two independent failure modes — retrieval failures (correct context not retrieved) and generation failures (correct context retrieved, wrong answer generated).

Separate evals let you fix the right thing. If retrieval recall is 60% and generation accuracy given correct context is 90%, your problem is retrieval. Improving your prompt will not fix it. If retrieval recall is 90% and generation accuracy is 70%, you have a generation problem. Adding more retrieval mechanisms will not fix it.

Build a retrieval eval that measures recall at k (did the relevant chunk appear in the top-k results?), precision at k (how many of the top-k results were relevant?), and mean reciprocal rank. Run it independently of your generation pipeline. Fix retrieval first, then generation.

What production looks like

A production search and retrieval stack at the systems we've built looks roughly like this: a query classifier, three retrieval backends (dense, sparse, structured), a re-ranker, a context assembler that handles deduplication and metadata threading, and a generation step. Each component is independently testable, independently deployable, and independently observable.

It's more complex than a single vector database and a similarity search call. It also works.