Why Raw Queries Break RAG

Here’s the pitch for RAG in one sentence: embed the user’s question, search a vector store for the closest matching chunks, hand those chunks to an LLM, get a grounded answer back. Clean, mechanical, almost too simple to fail.

So why do RAG systems keep retrieving the wrong thing?

The pitch skips a step it shouldn’t. Before you can ask whether retrieval “worked,” you have to ask what the system was actually comparing — and once you look closely at that comparison, a raw, naturally-typed question turns out to be a strange thing to search with.

What is a RAG system actually comparing?

An embedding model turns text into a vector — a long list of numbers that represents that text’s meaning as a point in a high-dimensional space. Every chunk in your document store was embedded once, ahead of time, during indexing. When a query arrives, it gets embedded the same way, and the system finds chunks whose vectors sit closest to the query’s vector — usually measured by cosine similarity, a way of scoring how much two vectors point in the same direction.

That’s the whole mechanism. It’s also where the trouble starts, because the chunks and the query were written by different people, at different times, for different purposes. A document chunk is a self-contained statement, written carefully, once, by someone who wasn’t anticipating your specific question. A query is typed in the moment, often mid-conversation, leaning on context the speaker assumes you already have. Feeding both into the same similarity function assumes they’re comparable objects. They usually aren’t.

That mismatch doesn’t show up as one problem. It shows up as several, each with a different shape — which is why there isn’t one fix called “query processing,” there are several, and each one exists because of a specific way the mismatch breaks retrieval.

What happens when the answer isn’t in the words you typed?

Picture a chat where someone asks “What was our marketing spend in Q4 2022?”, gets an answer, and follows up with “what about the 2023 numbers?”

Embed that follow-up exactly as typed and search with it, and you’re asking your index to find chunks about “2023 numbers” — numbers of what? From where? The word “numbers” isn’t wrong, it’s just incomplete. The missing piece — “marketing spend” — was never in this message. It’s sitting one turn back, in the conversation itself.

This is why most production RAG chat systems run an LLM call before retrieval even starts: take the current message plus the recent conversation history, and rewrite the current message into something that stands on its own — “What was our marketing spend in 2023?” — before it ever gets embedded. This is usually called contextualization (or conversational query rewriting). It doesn’t add new information the user never provided; it just relocates information that already exists in the conversation to where the retriever can see it.

What happens when one query is actually five?

Now take a different kind of question: “Compare the marketing budgets of all three regional offices in 2022 and 2023.”

Rewriting won’t help here — nothing is missing or ambiguous. The problem is structural. This single sentence is actually asking for six distinct facts (three offices × two years), and you’re about to represent all six of them with one embedding vector.

Think of a dense embedding as a single arrow fired from a single bow — it can point in exactly one direction. Ask it to hit six different targets at once, and it doesn’t hit any of them cleanly. It lands somewhere in the average of all six: a chunk that’s vaguely about budgets, vaguely about some offices, vaguely about some years, and precisely about nothing. That’s not a bad embedding model failing — it’s what a single vector is structurally incapable of doing.

The fix is to stop asking one arrow to hit six targets, and fire six arrows instead:

query = "Compare marketing budgets across 3 offices in 2022 and 2023"

sub_queries = [
  "Delhi office marketing budget 2022",
  "Delhi office marketing budget 2023",
  "Mumbai office marketing budget 2022",
  "Mumbai office marketing budget 2023",
  "Chennai office marketing budget 2022",
  "Chennai office marketing budget 2023",
]

results = merge_and_dedupe(retrieve(q) for q in sub_queries)

This is decomposition — an LLM splits the bundled query into sub-queries, each gets its own retrieval pass, and the results get merged. It’s solving a coverage problem: there are genuinely multiple facts to find, so there have to be multiple lookups.

What happens when you and the document don’t speak the same language?

Here’s a query that’s neither incomplete nor bundled: “why does my app crash on launch?” One fact, clearly asked. And yet the retriever might still come up empty, because the documentation chunk that answers it is titled “resolving startup initialization failures.”

A person reads both phrases and instantly knows they mean the same thing. An embedding model has no such guarantee. It places phrases near each other in vector space based on patterns learned from training data — and “crashing on launch” is colloquial, the way a user describes a problem, while “startup initialization failures” is formal, the way documentation describes the same problem. Register and vocabulary can pull two phrases apart in embedding space even when their meaning is identical. This is a lexical gap — not missing information, not a bundled query, just a mismatch in the words used to say the same thing.

You can’t know in advance which exact phrasing the right document uses. So instead of betting on one phrasing, you generate several and search with all of them:

query = "why does my app crash on launch?"

variants = [
  "why does my app crash on launch?",
  "resolving startup initialization failures",
  "app fails to start error troubleshooting",
]

results = merge_and_dedupe(retrieve(v) for v in variants)

This is expansion (sometimes called multi-query retrieval). Notice it looks almost identical to decomposition’s code above — multiple queries, multiple retrieval passes, one merge step — but the goal is the opposite. Decomposition fires multiple arrows because there are multiple targets. Expansion fires multiple arrows at the same target, because it isn’t sure which angle will land. One is a coverage problem; the other is a recall problem.

What happens when the right words point to the wrong moment in time?

Last shape of mismatch, and it’s the strangest one: a query and a document can match perfectly on meaning and still be wrong.

Someone asks “how many days of parental leave do I get?” Your index has a chunk on parental leave policy that is topically an exact match — same subject, same vocabulary, high cosine similarity. It’s also two policy revisions out of date.

No rewrite fixes this. You can rephrase the query as many ways as you like and never encode “must be the current version” into it, because recency isn’t a property of what the text means — it’s metadata about the document, sitting outside the content the embedding model ever sees. The fix has to live outside the embedding comparison entirely: a structured filter, applied alongside the semantic search — doc_date >= 2024-01-01 — that only lets current documents through, regardless of how similar an older one scores.

This is metadata filtering, and it’s a reminder that not every retrieval failure is really about the query at all. Sometimes the query was fine, and the index needed a second, non-semantic signal to lean on.

Should every query get the full treatment?

Four techniques, four distinct fixes for four distinct mismatches. The tempting move is to run all four on every query, every time, just to be safe. Resist it.

Each technique costs an LLM call before retrieval even starts — latency the user is waiting through — and each one is also a guess. Rewriting a query that was already fine risks introducing drift. Decomposing a query that only needed one lookup wastes five retrieval passes for nothing. Running everything, always, doesn’t just cost more — it can actively degrade queries that didn’t have a problem to begin with.

So production systems add a cheap classification step first: a small, fast model (not the expensive one doing generation) looks at the query and outputs one label — direct, contextualize, decompose, expand, or some combination with filter — and only that path gets executed.

label = classify(query, history)   # small/fast model, one token out

if label == "direct":
    results = retrieve(query)
elif label == "contextualize":
    results = retrieve(rewrite(query, history))
elif label == "decompose":
    results = merge(retrieve(q) for q in split(query))
elif label == "expand":
    results = merge(retrieve(v) for v in paraphrase(query))

The economics work out because most queries, in practice, don’t need heavy processing. If 70% of incoming queries are simple, self-contained lookups, a system that classifies-then-routes pays a small fixed cost on all 1,000 queries but only pays the expensive multi-pass cost on the 300 that actually need it — instead of paying the expensive cost on all 1,000 regardless.

The signals a classifier looks for are often visible on the surface, without needing a model at all: comparative words like “compare,” “versus,” “across,” or multiple named entities and dates in one sentence, point toward decomposition. Pronouns and backward references — “it,” “that,” “what about” — with no clear subject, point toward contextualization. A lexical gap is the hardest to catch this way, since there’s often no surface cue at all that the user’s words and the document’s words differ — which is exactly why some systems use a small LLM as the classifier instead of pure rules: it can reason about domain-vocabulary mismatch in a way keyword matching can’t.

What happens when the fix itself is wrong?

Every technique above shares an uncomfortable property: it’s an LLM guessing at intent before the retrieval system has shown it a single real document. What happens when that guess is wrong?

Nothing catches it. The pipeline runs in one direction — rewrite, then retrieve, then generate — with no step that checks whether the rewrite was faithful to what the user actually meant. A bad rewrite doesn’t fail loudly; it just quietly retrieves the wrong chunks, and the generation step answers confidently from the wrong evidence.

Decomposition and expansion each add their own version of this risk. Split a query into ten sub-queries and you get ten chunk sets handed to the generation model at once — and LLMs are well known to under-attend to information buried in the middle of a long context, a pattern usually called the lost-in-the-middle problem. The more you decompose, the more likely the one chunk that actually answers the question gets lost among the other nine. Expansion has a quieter failure: if four of five paraphrased variants are good and one drifts slightly off-meaning, the chunks it pulls in get merged into the result set with no flag marking them as coming from a shaky variant. The generation model treats them as equally valid evidence.

None of these are edge cases you can rule out by being careful once. They’re structural — every technique that fixes a retrieval problem does so by inserting a guess, and every guess can be wrong in a way nothing downstream is built to notice.

So where does that leave you?

The honest answer is: mostly with evaluation, not real-time correction. Metrics like recall@k (did the right chunks make it into the top results?) and faithfulness (is the generated answer actually grounded in what was retrieved?) can tell you, across many queries, that something in the pipeline is systematically off. What they can’t do is catch one specific bad rewrite, on one specific query, before it reaches the user — that would require validating the rewrite against the user’s original intent in real time, and the original query is often exactly the thing that was too incomplete or ambiguous to trust as ground truth in the first place. Comparing a rewrite’s retrieval results against a known-good answer only works if you already have the known-good answer — which is precisely what you don’t have, for a query nobody has seen before.

There is a class of architecture built specifically to close this gap — usually grouped under the name Agentic RAG — where the pipeline can inspect its own retrieved chunks, decide they don’t look right, and loop back to re-retrieve or re-process before ever generating an answer. It’s a real answer to the online-correction problem this post keeps circling. It’s also not free: a self-correcting loop is, by definition, more LLM calls sitting on the critical path of every query, which means more latency and more cost precisely on the queries that already triggered the most processing. Whether that trade is worth it, and how you’d design the loop so it doesn’t just add cost without fixing anything, is enough for its own post.

So the question worth sitting with isn’t “which technique should I add next.” It’s this: every one of these fixes makes retrieval better on average by making one part of the pipeline smarter and more autonomous — and every part that gets smarter and more autonomous is also a part that can now fail silently, in a way nothing else in the system is watching for. At what point does a RAG pipeline’s growing intelligence start costing you more in undetectable failure modes than it earns you in retrieval quality — and how would you even know if it already had?

What’s your take? Drop a comment below.