The demo works. Somebody asks how you know it will keep working after the next prompt change, and the room goes quiet. That silence is the whole problem, and it has a cheap fix that almost nobody applies early enough.
We have argued in two previous articles that this matters — that evaluation is the cost nobody budgets, and that you cannot tune a retrieval parameter you cannot measure. Neither said how. This one does.
Why "it looks good" stops working
Manual checking scales to about the first twenty examples. Past that, three things break at once: nobody re-checks the cases that already passed, so regressions land silently; different people judge differently, so results are not comparable week to week; and the person checking has seen the prompt, so they read the output charitably.
The failure mode this produces is specific and recognisable. A prompt change fixes the complaint that prompted it and quietly breaks two behaviours nobody was watching. Without a suite, you find out from a customer, roughly a month later, and you cannot tell which change did it.
The compounding cost is worse than the incident. Once a team has been burned twice, it stops changing the prompt — and a prompt nobody dares touch is a frozen product with a model provider deprecating underneath it. The eval suite is not really about quality assurance. It is about retaining the ability to change anything at all.
Start with fifty examples, not a framework
The instinct is to choose an evaluation library first. Resist it for a week. The library is interchangeable and takes an afternoon to swap; the examples are the asset, and they are the part only your team can produce.
Fifty cases is enough to be useful and small enough to finish. Draw them from real user questions, not invented ones, and weight them towards the places you already know are hard: the ambiguous phrasing, the question whose answer changed last quarter, the one where the correct response is to refuse.
- Write the negative assertion. What must not appear is usually more testable than what must. "Does not mention the 30-day window" catches the exact regression you fear.
- Include the refusals. A quarter of your set should be questions where "I don't know" is the correct answer. Systems that never refuse score well right up until they are wrong in public.
- Date every case. Without an as-of date, a case about current policy silently becomes wrong when the policy changes, and you will spend an afternoon debugging a model that is behaving correctly.
Three kinds of check, in increasing cost
Most teams reach straight for the middle tier because it feels like the sophisticated answer. The bottom tier is where the return is. Deterministic assertions are free to run, produce no false confidence, and catch the majority of regressions that actually reach production — malformed JSON, a missing citation, a leaked email address, an answer four times longer than the UI can display.
A useful rule: if a check can be written as assert, it belongs in the bottom tier and should never be handed to a model. Reserve the judge for the genuinely subjective question — is this answer helpful, is the tone right, does it actually address what was asked.
If you want the concrete starting list, these five deterministic checks earn their keep in almost every retrieval-backed feature, and all five are a morning's work: the response parses as the schema you promised the UI; every citation resolves to a document that was actually retrieved; no string from the forbidden list appears; no email address or identifier that was not in the input appears in the output; and the response fits the length the interface can render.
A citation that does not resolve is a hallucination you can catch with a database lookup.
Using a model as a judge, and what it costs you in trust
A model scoring another model's output against a written rubric is a genuinely useful technique. It is also the point at which your evaluation acquires a dependency that itself needs evaluating, and that second step is the one teams skip.
The largest systematic study to date evaluated 21 judges from nine providers and named the trap precisely: reliability without validity. Some judges show test–retest reliability above 0.95 — ask twice, get the same answer — while simultaneously exhibiting position bias above 0.10, meaning the winner changes when you swap which candidate is shown first. arXiv:2606.19544.
The number worth internalising is the kappa gap: between raw agreement and Cohen's kappa the study found a deflation of 33 to 41 percentage points on MT-Bench. Raw agreement counts the cases the judge got right by chance; kappa does not. A judge reported at "85% agreement with humans" may be doing very little better than a coin weighted to the common answer.
Validating the judge itself
RAND released a Judge Reliability Harness in March 2026 that treats a judge as a model-plus-rubric pair and stress-tests it. Evaluating four judges across safety, persuasion, misuse and agentic benchmarks, the authors concluded that no judge was uniformly reliable — consistency broke down on formatting changes, paraphrasing and shifts in verbosity.
You do not need that apparatus to get most of the protection. Three cheap habits cover the failure modes above.
- Have humans label thirty cases the judge has already scored, and measure agreement with Cohen's kappa rather than raw percentage. If kappa is under about 0.6, the judge is not ready to gate anything.
- Run every pairwise comparison in both orders and average. This costs double and removes position bias outright, which is the single largest measured effect.
- Re-run a fixed sample whenever you change the judge model or the rubric, and compare against the stored scores. A judge upgrade that silently rescores your history is indistinguishable from a product regression.
And keep the rubric short. A rubric with nine criteria produces judgements that correlate with nothing in particular; three concrete criteria with worked examples of a pass and a fail produce something you can argue with.
Wiring evals into CI
An eval suite that runs when somebody remembers is not a gate. The deterministic tier is fast and cheap enough to run on every commit; the judge tier costs real money and belongs on merge to main, or nightly if your volume makes even that expensive.
The threshold is where most teams get it wrong. An absolute bar — "must score above 90%" — either blocks everything on the day you add hard cases, or gets lowered until it never fires. Gate on the delta instead: fail if the pass rate drops more than two points below the last main build, and fail unconditionally if any case tagged critical regresses. That fires only on changes you caused.
Pin the judge model version explicitly. An unpinned judge means your baseline moves under you when the provider updates the model, and you will spend a day looking for a regression in your own code that is not there.
The running cost is smaller than people fear and worth stating plainly. Two hundred cases through a mid-tier model, with both orderings for the pairwise comparisons, is a few euros per full run — well under the cost of the engineer waiting for it. Use a cheap model for the deterministic tier's occasional judgement calls and reserve the expensive one for the rubric, and run the whole suite nightly rather than blocking on it if latency is the constraint. The reason teams do not run evals is almost never the token bill.
What to measure once it is live
The eval set built before launch measures the failures you could imagine. Production measures the ones you could not, and the gap between those two sets is where the interesting work is. Every complaint, every thumbs-down, every support ticket that starts "the assistant told me" is a case waiting to be written down.
- Refusal rate. Trending to zero usually means the system has learned to answer things it should decline, not that it got smarter.
- Retrieval-empty rate. How often the model answered with nothing retrieved. Those answers came from the weights, not your documents, and they are your hallucination risk concentrated in one metric.
- Human override rate. Where a reviewer is in the loop, this is your most honest quality signal, and it is free.
One caution about the loop: adding a case for every complaint eventually turns the suite into a museum of past incidents, weighted towards whatever went wrong loudest. Every quarter, re-read the set and ask whether it still resembles current traffic. Drop cases whose feature no longer exists. This is the same maintenance argument the AI Act makes for declared accuracy — a metric you can defend has to be measured on data that reflects what the system actually does today.
Where to start
- Write fifty cases this week. In a YAML file, in the repo, next to the code. No framework, no platform, no procurement.
- Automate the assertions first. Schema, forbidden strings, required citations. You will be surprised how many real regressions this alone catches.
- Add a judge only when you have a rubric. Three criteria, worked examples, and thirty human-labelled cases to check it against. Without those you have added a dependency, not a measurement.
- Gate on the delta, not an absolute. And pin the judge version, so the baseline means the same thing next month as it does today.
- Grow the suite only from real failures. One new case per production complaint. A suite that grows any other way is measuring your imagination.
None of this requires a platform, and the whole first pass fits in an afternoon. The reason it rarely happens is that it produces no demo — only the ability to change something next month and know, before a customer tells you, whether you broke it.
