The Tests Kept Passing

The descriptions keep sounding familiar.

Someone writes about an eval set that has stopped catching regressions. Someone else describes a judge model that agrees with itself on every run and is wrong in the same direction on every run. The two problems have nothing to do with each other, and both read like something already lived through.

There was no particular article where that landed, no conversation that set it off. It accumulated, the way these things do, until the recognition got frequent enough to be worth distrusting.

Because recognition is cheap. Two things can look alike because they share a mechanism, or they can look alike because eighteen years in software - quality engineering for most of it, application development and tech leadership alongside - is long enough to find a familiar shape in almost anything. The first case is useful. The second is pattern-matching dressed up as insight.

So the map below comes with a caveat attached, and then three attempts to break it, using failures there was no one else to blame for.

Does the vocabulary actually line up?

Here’s the map, before any of the caveats:

The left column is thirty years old. The right column is about three. If the third column is genuinely the same question in both cases, then a fair amount of hard-won practice transfers, and the AI evaluation conversation is rediscovering things that are already written down.

That is a very convenient conclusion for anyone with a testing background, which is exactly why it needs pushing on. Three rows, worked properly.

What happens when the world moves and the suite doesn’t?

Sometime back, I was tech lead on a system with an integration at its centre. The unit tests passed. The mocked integration tests passed. The real integration, when we stood it up, worked.

We had not anticipated every use case. Nobody does. But the specific gap mattered: at some point after we shipped, the upstream system started sending non-ASCII characters where it previously had not. Our system dropped the characters it couldn’t match. It didn’t error. It didn’t warn. It processed the record, wrote the result, and moved on.

Customers were affected unevenly, because whether the dropped characters mattered depended on other parameters interacting with them. Some saw nothing wrong. Some saw something subtly wrong and didn’t report it. It surfaced during a monthly audit.

The tests passed the entire time.

Notice what kind of failure that is. The suite wasn’t wrong when it was written - it was a reasonable suite for the system as it existed. It stopped being adequate without changing at all, because the thing it was testing against changed underneath it. Nothing in the codebase moved. Nothing in the test suite moved. The world moved, and the suite kept reporting on a world that no longer existed.

Now put an eval set in the same position. You build it against a sample of production traffic. It’s a good sample - genuinely representative, on the day you build it. Then your users change. A new customer segment arrives with different phrasing. A product change routes different questions into the system. The distribution your eval set was drawn from stops being the distribution you’re serving.

Your eval score doesn’t move. It can’t. It’s still measuring the old distribution faithfully, which is precisely the problem.

This row holds, and it holds more tightly than expected. The mechanism is the same in both fields: a frozen measurement against a moving target, reporting health it has no way to verify. The only real difference is one of rate. Input distributions to an LLM system drift faster and less visibly than upstream contracts in traditional integration work, where at least somebody usually announces a schema change. But that’s a difference of degree, and it makes the eval case worse, not different.

Why didn’t anything complain?

That’s the part of the story worth sitting with. Not that there was a gap - that the gap produced no signal.

There’s a version of that failure which is easy. The upstream sends something unexpected, the parser throws, the alert fires, someone gets paged, it’s fixed inside an hour. That’s a good failure. It’s loud and it’s cheap.

What happened instead was a system producing plausible output. Records went through. Fields were populated. Nothing in the pipeline was constructed to say “the thing I just wrote is missing characters that were in the thing I read.” The absence was the defect, and absence is exactly what a system doesn’t notice unless you build it to.

This is the most suspect row on the table, because “hallucination is like a test passing for the wrong reason” sounds like the kind of analogy that flatters whoever makes it. But it survives, and the reason is specific: in both cases the output is well-formed. A hallucinated answer is fluent, correctly structured, confident. A green test result is a green test result. In neither case does the artifact you’re looking at carry any signal about whether it’s trustworthy. You have to go outside it to find out.

RAGTruth, a public hallucination corpus, is useful here precisely because it was built by going outside it: human annotators marking hallucinated spans, character offset by character offset, across nearly eighteen thousand responses. The reason that dataset was expensive to build is the same reason an audit caught what the tests didn’t. Somebody had to look at the actual content, against the actual source, with no shortcut available.

Where the row weakens: a test passing for the wrong reason is a defect in the test. Someone wrote an assertion that doesn’t assert what they thought. It’s fixable, and once fixed it stays fixed. A hallucination isn’t a defect in the eval - it’s the system doing the thing it does. You can catch more of them; you can’t write an assertion that makes the model stop generating them. The analogy describes the shape of the failure well and the remedy not at all.

Who was checking the checker?

Here’s the detail from that incident that took longest to appreciate, and it’s the one most worth taking away.

The mocked integration test passed. Of course it did. We wrote the mock. We wrote it based on our understanding of what the upstream system sent, and our understanding was the same understanding that produced the parsing code. The mock and the code were built by the same people, from the same assumption, on the same day.

So the mock could confirm that the code did what we thought. It had no way to tell us that what we thought was wrong. It wasn’t an independent check. It was our own assumption, restated in a second file, agreeing with itself.

A few weeks ago, writing about system prompts here, the same point came up from the other direction - a safety instruction living in the same generation pass as the action it governs isn’t a check, it’s a second opinion from someone who already agreed with the first opinion. What didn’t register at the time is that the exact problem had already turned up years earlier, in a mock.

That’s the test-oracle row, and it’s the one where AI evaluation looks to be in genuinely worse shape than testing ever was. An LLM judge is an oracle assembled from roughly the same training distribution, and often literally the same model family, as the system it’s judging. If the generator has a blind spot that comes from its training data, there’s no reason to expect the judge not to share it. The judge can be perfectly consistent and still be structurally incapable of seeing the one thing you most need caught.

There’s published work pointing at this, and the title does most of the work: Reliability without Validity, a systematic evaluation of twenty-one judges from nine providers across three benchmarks, published in June. Among its findings - two of those judges, both deployed in production, combined test-retest reliability above 0.95 with severe position bias. Ask either one the same question twice and it answers the same way. Change the order the two candidate answers are presented in, and it changes which one wins.

Anyone who has spent time on measurement systems will recognise those as the two separate properties they are. Reliable means repeatable. Valid means measuring the thing you meant to measure. They are not the same property and one does not imply the other - a sentence that sounds obvious right up until you’re looking at a consistent green result and deciding whether to trust it.

Two out of twenty-one is not an epidemic and shouldn’t be inflated into one. What makes it worth noting is that repeatability is the property you’d naturally check first, it’s the cheap one to check, and in those two cases it was the property that would have reassured you.

The difference is that with a flaky-looking test you at least get a hint something’s off. With a consistent one you get nothing. You get a green tick and a clean conscience.

What do you do with a signal you can’t trust?

Which brings up the least comfortable part.

We had a flaky suite. The immediate response - and this was our response, not one watched from a distance - is to re-run it and hope the pass count changes. Then you take the union of the successes across both runs, and you generate the report by hand.

Sit with what that actually is for a second. Run one: these fail. Run two: those fail. Union of passes: everything passed. The report says the suite is green. No single run of that suite was ever green. We produced a result that had not happened.

It happened more than once, under delivery pressure, and it involved enough manual work and enough human judgement that we should have noticed what we were doing sooner than we did. Eventually we did notice - mostly by admitting that we weren’t solving a problem, we were scheduling one.

So we quarantined the failing cases. That is not a free move: it immediately increased the manual testing burden, because the things we’d stopped asserting automatically still needed somebody to check. But it bought a suite whose green meant something. Then the quarantined cases got worked properly - planned, fixed, run under varied conditions until they passed for real rather than passing when we were lucky. In the UI tests, some of it was less principled: hard-coded waits, more automatic retries.

Now: how much of that transfers to an unreliable LLM judge?

The first part transfers uncomfortably well. Running an eval repeatedly and reporting the best number is the same move as that union of successes, and it’s easier to do and harder to spot, because nobody hand-assembles anything - you just re-run and keep the output you liked. Most people doing it probably wouldn’t recognise it as the same move. We didn’t, at the time.

The quarantine step transfers too, with its cost intact. Dropping noisy cases from your headline metric buys a number you can trust across a smaller surface, and you pay for it in the things you’re no longer watching. Same trade, same rule attached: it’s only honest if the quarantine comes with a plan.

And the hard-coded waits have an analogue that is genuinely uncomfortable. Tightening a judge’s rubric, or constraining output format, until agreement improves - that reduces the variance you can measure without necessarily improving the system underneath. We did that to UI tests because the deadline was real. The same pressure probably produces the same move in eval work, for the same reasons, with the same self-deception attached.

Does the fix transfer, though?

No. This is where the row breaks, and it’s the most interesting break on the list.

In testing, flakiness is a defect. That’s the whole premise of the approach above - quarantine is a holding pattern, the real work is the fix, and there is always a fix, because the non-determinism is coming from somewhere: a race, a shared fixture, a timing assumption, an environment difference. Find it, remove it, the test becomes deterministic, it stays deterministic.

An LLM judge has no such somewhere. The variance isn’t leaking in from a badly isolated fixture. It’s the mechanism. Sampling from a distribution is what the thing does, and you cannot fix it without replacing it with something that isn’t a language model.

So the playbook loses its ending. Quarantine with no prospect of repair isn’t a holding pattern, it’s just a permanent blind spot everyone has agreed not to mention. “Run it under different conditions until it passes” stops being diagnosis and becomes tampering. The moves are the same; the thing they were moves toward is gone.

Which leaves an open question, and it’s the one currently getting worked on: if the flakiness can’t be eliminated, how many runs does it take before a change in an eval score means something real rather than the instrument moving? Testing has a well-developed answer to “is this failure real” when the flake is fixable. There doesn’t appear to be one for when it isn’t. That measurement is underway, and whatever it turns up is worth publishing either way - which beats asserting the analogy holds and leaving it there.

What else doesn’t survive the crossing?

Three more places the map runs out, briefly, because they’re worth naming even without stories attached.

There’s no coverage equivalent. Code coverage is imperfect and gameable and still enormously useful, because it answers “what haven’t we looked at” with a number. “Have we evaluated this system’s behaviour space” has no such number, and doesn’t look close to having one. This is where testing has the least to offer and the most to learn.

The system under test can change underneath you. In testing, the build is pinned. That isn’t a nice property, it’s the premise - you cannot attribute a behaviour change to your change if the thing you’re testing moved on its own. A hosted model can update with no version bump you control and no changelog you were shown. There’s no traditional testing analogue for that at all, and of everything on this list it’s the one that looks genuinely new rather than renamed.

Re-running costs money. “Run it a hundred times and see what survives” is free in CI, which is why it became the reflex. At API prices per run, against a real eval set, it’s a budget conversation. The technique crosses over. The economics that made it obvious don’t.

Where this leaves you

The vocabulary lines up better than expected. The remedies line up much worse.

Every row in that table describes a real shared question, and a quality engineer reading about eval work is right to feel they’ve met these problems before - they have. But the thing testing actually accumulated over thirty years wasn’t a vocabulary. It was a set of fixes, and the fixes assume determinism the way a fish assumes water: the build is pinned, the flake has a root cause, the assertion can be corrected, and a green result that lies is a bug rather than a property. Take those assumptions away and a fair amount of the practice is left holding a diagnosis with nothing to prescribe.

The four rows left unworked here - guardrails against input validation, red-teaming against negative testing, prompt regression against config drift, suite rot against eval drift - probably hold too, with less confidence behind them and no first-hand evidence. Better to say that plainly than imply eight worked arguments where there are three.

Which makes the direction of the lesson the opposite of the expected one. This isn’t an older discipline arriving with answers for a younger one. It’s an older discipline arriving with a good set of questions and a toolbox that half fits.

And that leaves one thing still turning over. Every failure described here was eventually caught by something outside the automated suite - an audit, a customer, a person who went and looked. If the new failures are quieter than the old ones, and the instruments for catching them are themselves probabilistic, what is the modern equivalent of the monthly audit - and is anyone building it, or is it being assumed that the evals have it covered?

What’s your take? Drop a comment below.