Graph RAG Is Not Better RAG

Picture a small experiment. You have a handful of short documents about drug interactions - except the drugs are invented. Not real names: Faxiprine, Tricalamine, Renal Shimmer, Hydrolyx Draught. That matters, because if the drugs were real, a language model could answer from what it already learned in training and we’d never know whether retrieval was doing any work. Fictional drugs force every answer to come from the documents in front of it. Nowhere else.

The documents are written so that no single one contains a whole story. The facts are deliberately scattered:

Document A  →  "Faxiprine and Tricalamine interact when taken together."
Document B  →  "Patients who combine Faxiprine and Tricalamine develop Renal Shimmer."
Document C  →  "Hydrolyx Draught is a remedy that treats Renal Shimmer."
                                                   ↑
                              Document C never mentions Faxiprine or Tricalamine.

Now ask the obvious question: “If I take Faxiprine and Tricalamine together, what remedy addresses the resulting side effect?” The answer is Hydrolyx Draught. It’s right there in Document C. And here is what an ordinary retrieval system said:

“There is no information about a specific remedy that addresses side effects resulting from taking Faxiprine and Tricalamine together.”

Every fact it needed was in the corpus, and it came back empty-handed. This isn’t about a weak model. It’s about a shape of question that ordinary retrieval cannot answer, no matter how good the embeddings are. Understanding why is the fastest way to see what Graph RAG actually buys - and what it charges.

Why does similarity search go blind here?

Start with how ordinary RAG - “vanilla” RAG - finds anything.

You cut your documents into chunks and turn each chunk into an embedding: a long list of numbers that captures its meaning, arranged so that chunks about similar things sit near each other in that number-space. Text about kidney side effects lands near other text about kidney side effects. At query time you embed the question, grab the handful of chunks sitting closest to it, and hand those to the model. That’s the whole mechanism. Retrieval is nearness in meaning-space.

Look again at the scattered documents. The answer needs a chain of three facts, and each fact lives in a different document. Fine so far. But notice Document C - the one that actually holds the remedy. It talks about Hydrolyx Draught and Renal Shimmer. It never mentions Faxiprine or Tricalamine. So when you embed a question full of drug names, Document C is nowhere near it in meaning-space. Nothing pulls it into the top results. The one document with the answer is invisible to the only tool vanilla RAG has.

That’s the structural blind spot: similarity search measures whether two things are alike, never whether they are connected. A single-hop question (“What does Hydrolyx Draught treat?”) works, because question and answer sit in the same chunk and land near each other. A multi-hop question forces you to connect chunks that are unlike the question and unlike each other. Nearness can’t do that. And it gets worse with distance - each extra hop is another chance for the chain to fall out of the top results entirely.

What does a graph add that a pile of chunks can’t?

Imagine navigating a city with a stack of landmark photos. Ask “how do I get from the museum to the harbour?” and the stack hands you its best matches: a nice shot of the museum, a nice one of the harbour. Both relevant. Neither tells you they’re a twenty-minute walk apart, or that the route runs past the station. Photos capture places. They say nothing about what connects them.

A map is the same landmarks with the roads drawn in. You don’t just locate the museum - you trace the line from it, through the streets, to the harbour, even though no single photo ever showed both. That drawn-in road is the primitive a pile of chunks structurally lacks. Vanilla RAG is the photo stack. A knowledge graph is the map.

On the map, the landmarks are nodes and the roads are edges - labelled relationships. Our scattered facts become one connected path:

Faxiprine ──INTERACTS_WITH──▶ Tricalamine ──CAUSES──▶ Renal Shimmer ──TREATED_BY──▶ Hydrolyx Draught

The one thing edges give you that nearness never can is traversal: start at a node and walk the connections to reach another, following a path instead of measuring a distance. “A connects to B, and B connects to C” is a sentence a map can express and a stack of photos cannot.

How does Graph RAG use the map at query time?

Watch the same question run through the graph pipeline. It moves in three steps, and none of them is “grab the nearest chunks.”

First, entity resolution. Pull the real things out of the question - Faxiprine, Tricalamine - and match them to nodes on the map.

Second, traversal. Starting from those nodes, walk the edges until you reach an answer:

Faxiprine ─INTERACTS_WITH─▶ Tricalamine ─CAUSES─▶ Renal Shimmer ─TREATED_BY─▶ Hydrolyx Draught
   └────────────────────────── the path the query walks ──────────────────────────┘

Third - the step people miss - targeted retrieval. The graph is not the answer. Each edge carries a pointer back to the document it came from. Having walked the path, the system fetches exactly those chunks - including Document C, which similarity search could never surface, because the map led straight to it regardless of its wording.

Same corpus, same model, only the retrieval changed:

“If you take Faxiprine and Tricalamine together, the resulting side effect is Renal Shimmer, which is addressed by Hydrolyx Draught.”

And notice what the third step preserves: the map didn’t replace the documents. It’s an index over them - roads drawn on top of the photos, not instead of them. The model still reads the real source text; the graph only decides which text is worth reading. Hold onto that. It comes back.

But how do you build this map on a real corpus?

It’s tempting to picture construction as a tidy sequence: load document one, create its nodes and edges; load document two, find a node that already exists, and extend it. That’s fine for the handful of documents in our example, which fit in memory. It quietly breaks at sixteen million.

At scale, construction splits into two phases:

Doc 1 ─┐                                  ┌─▶ (faxiprine, INTERACTS_WITH, tricalamine)
Doc 2 ─┤   PHASE 1: extract in parallel   ├─▶ (tricalamine, CAUSES, renal_shimmer)
Doc 3 ─┼──  each document read alone,  ───┤   (renal_shimmer, TREATED_BY, hydrolyx…)
 ...   │    no shared graph yet            └─▶ (faxiprine, INTERACTS_WITH, tricalamine)
Doc N ─┘                                              │  loose triples, tagged with source
                                                      ▼
                                    PHASE 2: merge by normalized name
                                                      │
                                                      ▼
                              one node per key, every matching edge attached

Phase one, extraction, is deliberately blind. Every chunk is read on its own, in parallel, knowing nothing about any other chunk. A model reads a chunk and emits triples - (subject, relationship, object) - each tagged with the chunk it came from. No shared graph exists yet. That blindness is the trick: because extraction doesn’t need the growing graph, you can fan it across hundreds of workers instead of trudging document by document.

Phase two, the merge, is where the map is actually drawn. All those loose triples fold together, joined by a key - a normalized version of the entity name (lowercased, whitespace stripped). Every triple mentioning faxiprine, from any of the sixteen million documents, collapses onto one node because they share that key. So “how does document two’s fact attach to document one’s node?” isn’t “we found it in memory” - it’s both were extracted blind, and they met on the same node because their names normalized to the same key. A GROUP BY, not a walk.

Your sequential “extend the existing node” intuition isn’t wrong - it’s just a different job. That’s the incremental update path: once the graph exists, a new document is extracted and merged in by the same key, so familiar names extend existing nodes and new ones appear. Construction is the bulk build; your intuition is the update.

And that key is the whole ballgame. In our toy, “same entity” is trivial. On a real corpus, matching by normalized name is blunt: it leaves Jon and Jon Márquez as two people because the strings differ, and merges two different Springfields because they don’t. Getting this right - entity resolution - runs from cheap string normalization, to grouping similar names and fuzzy-matching within each group, to asking a model “are these the same thing?” for the hard cases. Remember it. It’s about to matter.

What does the map cost - and how does it lie?

Two prices. One obvious, one nasty.

The obvious one: a graph is never finished. A vanilla index grows the easy way - new document, embed, insert, done. A graph must be re-extracted and re-merged as documents change. Standing cost, not a one-time build.

The nasty one is a new kind of wrong, and it comes from the map itself. Extraction reads a sentence and draws a relationship - and it can draw the wrong one: label a CAUSES as a TREATS, point an arrow backwards, attach a fact to the wrong node. Suppose just one edge on our map comes out inverted. The truth is that combining the two drugs causes the side effect, but the graph records the opposite:

the truth:   Faxiprine + Tricalamine ──CAUSES──▶ Renal Shimmer
the graph:   Tricalamine ─────────────TREATS───▶ Renal Shimmer

Now someone asks: “Does Tricalamine treat Renal Shimmer, or cause it?” When similarity search fails, it fails loudly - irrelevant chunks, a hedged answer, an audible “I don’t know.” A wrong edge fails silently: the graph is trusted as ground truth, so traversal walks the false edge and the model answers, confidently, “treats.” One bad edge poisons every question that ever crosses it. A wrong road on a map is far more dangerous than a blurry photo - the photo you distrust, the map you follow.

There’s a tempting escape hatch here, and it’s worth dismantling because it will occur to you. Remember that Graph RAG doesn’t only hand the model the edge - targeted retrieval also pulls the source document that edge came from. So the model might read the original text, notice it actually describes the drugs causing the condition, and flag the contradiction on its own. Sometimes it will.

Do not build on that. It is not a safety net, for two reasons. First, it depends on the model choosing to doubt a structure it was handed as authoritative. Instruct it firmly to trust the graph, or swap in a terser model, and it follows the edge without a second glance. Second, and far worse: self-correction only has a chance when the mistake contradicts the text at all. The most dangerous structural errors don’t. Picture the build step deciding two different drugs are the same entity and collapsing them into a single node. Every document on the resulting path still reads as true on its own - each individual fact was correct; only the joining of them was wrong. There is no contradicting sentence anywhere for the model to catch. The map is confidently wrong, and every document stapled to it agrees.

So the honest lesson runs opposite to “the model will notice.” Because the graph points back to its source text, you always have the raw material to catch a bad edge - but the catching cannot be left to the model’s goodwill at answer time. It has to be a deliberate step earlier: check edges against their source when you build them, verify the relationship types you can’t afford to get wrong, sample for human review where a wrong answer is expensive. And this bites hardest exactly when you scale up - because at that size nobody mislabels edges by hand. Extraction and entity resolution do it for you, in bulk, on documents no one will ever read. That is where confident, well-reasoned, wrong Graph RAG answers are really born.

So when is the map worth drawing?

Not “when you can afford it.” The honest rule has two dials.

Query shape. If your questions are single-hop - answerable from one chunk - a graph buys almost nothing and charges plenty. The advantage is roughly zero at one hop and compounds with each additional hop. Graphs pay off when answers live in the connections between documents.

Cost of a confident lie. Because the failure is silent, the real question isn’t “will it ever be wrong” but “what does a wrong answer, delivered with total confidence, cost me here?” A recipe blog shrugs it off. Drug interactions, finance, and law cannot - and that’s exactly where the query-shape dial also points at graphs. High stakes justify the entity-resolution checks, the edge verification, the review loops. Low stakes don’t.

The reframe

Graph RAG is not the upgraded RAG you switch on once you can afford it. It’s a different point on a cost/risk curve. It buys traversal - answers hiding in the connections between documents, which similarity search structurally cannot reach. It charges an index that’s never finished, and a failure mode that reasons confidently to a wrong answer instead of failing out loud.

So before you reach for the map, ask the sharper question: for a corrupted edge you’ll never notice by hand - and a merged entity that leaves no contradiction in the text at all - what is your plan to catch it? If you have a good answer, Graph RAG earns its keep. If you don’t, you may just be building a faster way to be confidently wrong.

What’s your take? Drop a comment below.