The Model Writes, the Judge Measures: Anatomy of an LLM judge

By Sam Wen, CEO & Founder of XInfer.AI — July 23, 2026

Here is the kind of sentence that should keep a team up at night:

"At this clarity grade, you won't see a thing with the naked eye."

Our assistant produced it while explaining a diamond. The grade it cites is real — SI1, printed on the stone's lab report. The promise attached to it is invented. A clarity grade is assigned under 10× magnification; it does not resolve what a naked eye will see. Two SI1 stones can look entirely different — one clean at arm's length, the other with a visible speck right in the middle. The model took a true fact and manufactured its consequence, because that is what a helpful sentence sounds like.

Notice what the sentence is not. It is not a banned word. It is not a hallucinated stone. It contains no wrong number. It breaks no rule you could have written down in advance. It is simply fluent — and fluency is the problem. Fluent and true are different variables, and a language model optimizes the first one.

We build AI assistants for exactly this kind of product. The one in this essay explains diamonds: a short, two-voice spoken conversation about one stone — a host asks the questions a shopper would ask, a gemologist answers — generated from that stone's evidence. A diamond is the definition of a high-consideration purchase: thousands of dollars, bought rarely, chosen under uncertainty. And it has a property that makes it an unforgiving test bed for generative AI — nearly every claim about a diamond can be checked against its certificate from GIA or IGI, the independent laboratories that grade the world's diamonds. Overselling is not just tacky here. It is checkable.

In an earlier essay we argued that the language model should conduct the conversation while a deterministic core makes every decision. This essay is the same worldview applied to quality. The model writes. It does not get to grade.

At one generation you can catch a sentence like the one above by reading it. Across a catalogue of stones you cannot. So before we made the writer better, we built an instrument: an independent LLM judge that reads every generated claim against the evidence and returns a verdict we can count. This essay is about how that judge is built, how we keep the judge itself honest, and what changed once we had it — which is that the arguments stopped.

Part I — Three gates, and who owns which rule

The assistant explains one diamond at a time, from that diamond's evidence: the lab certificate, the price, the retailer's published analysis of the stone. One rule governs everything: every claim about this stone must trace to supplied evidence. If the evidence cannot answer — is it eye-clean, meaning no marks visible to the naked eye? — the assistant says so, out loud: the honest answer is a maybe, and it will not promise more. Then it points the shopper to a human gemologist.

Enforcing that rule is not one mechanism. It is three, layered, and they differ in kind.

The first gate prevents. It runs in code, inside the pipeline. When the assistant generates a follow-up question about a stone, the question must name the evidence path it would be answered from. A question that names no available path is rejected — by code, not by prompt hope. This is what stops the system from asking "how does this price compare to the market?" about a stone whose data has no comparable set. The model never gets the chance to improvise an answer, because the question never survives.

The second gate catches. A deterministic validator runs over every finished script: a hard-fail lexicon of the words the assistant must never say — amazing, must-have, investment, the manufactured urgency, the "trust me." Word-boundary regexes, a few dozen tests, no judgment involved. If a banned phrase appears, the script fails. Cheap, fast, and absolute.

The third gate judges. What remains after prevention and regex is the dangerous class: the plausible-but-unsupported claim. "You won't see a thing with the naked eye" passes both gates. No rule names it in advance, because it is not a phrase — it is a relationship between a sentence and a lab report. Checking it requires reading both. That is a job for a reader, and at scale the reader has to be a model.

One detail here taught us more than we expected: the boundary between the gates is itself a design decision. When we extended the banned-word list from the domain experts' style guide, the unambiguous terms went into the regex gate. But the context-dependent ones deliberately did not — and the best example is flawless. In most products, "flawless" is empty praise. In diamonds it is literally a grade: FL, the top of the GIA clarity scale. "It is not flawless" is a factual statement about an SI1 stone, not a compliment withheld. Likewise "guaranteed": a real test fixture proved that "that would need a guarantee the report doesn't give" is the assistant at its most honest. A blunt regex would fail exactly those sentences. Those words went to the judge instead, which can read the sense and not just the string. Which gate owns which rule is not a default. It is a decision, and we made it with test evidence.

Part II — Anatomy of a verdict

"Use an LLM judge" usually means: ask a model to rate the output from one to ten. A score like that is an opinion with a number attached. You cannot appeal it, you cannot audit it, and you cannot tell whether a 7 means two small problems or one large one. It is not an instrument; it is a mood.

Our judge is built differently, and every difference exists because a naive version failed first.

It returns a triage, not a score. For every sentence that asserts something beyond restating the input data, the judge answers typed questions: Is this a claim about this stone, or about diamonds in general? If it is about this stone — does the evidence establish it? If it is general — is it accurate gemology? "Lab-grown and natural diamonds are chemically identical" is category teaching: permitted, and wanted — the judge's instructions say teaching is never a violation. "This stone will look white to the eye" is an item claim, and it had better trace to something. A real violation is only the intersection: about this stone, and unsupported.

A claim also has an owner. Supported is not the only axis. GIA graded the stone; the retailer analyzed it; the assistant speaks for the house. If it says "we grade this VS1," every word is factually true and the sentence is still a violation — it credits the lab's work to the house. The judge flags misattribution as its own class, separate from unsupported claims, because provenance errors are how a trustworthy voice quietly claims authority it does not have.

It must report the claims it cleared. The judge is required to list the general statements and the supported claims too, with the classification visible — and it is forbidden from letting them inflate the violation count. This sounds like bureaucracy. It is the whole point. A verdict that only lists problems is a number you have to trust. A verdict that shows every ruling — each quote verbatim, each one naming the evidence path it was checked against — is an audit trail you can re-check. When the judge is wrong, you can see exactly where.

Orthogonal signals stay orthogonal. The judge also measures tone: does the assistant sound like a trusted expert across the counter or a closer working a commission? Does the conversation sound like two people responding to a stone, or a script? But tone is a separate, subordinate channel, and the judge's instructions are blunt: warmth never excuses an unsupported claim. A formatting check rides in a third channel and changes nothing in the honesty count. There is no blended score, deliberately. A blended score is where warmth hides dishonesty.

The writer and the judge read the same constitution. Every rule about what a lab report does and does not support lives in one content-standard document. That document is spliced into the writer's prompt and into the judge's prompt. The spec is a single artifact: the writer implements it, the judge enforces it, and editing one sentence changes both sides of the contract at once. There is no version of the rules that only the judge believes.

It never grades its own output. The writers run on Claude Sonnet 5. The judge's seat has been held by two models — first GPT-5.5, a different lab entirely, and today Claude Opus 4.8, a different model from the writers — and we verified the discipline held across the change. A model asked to grade its own writing has a conflict of interest baked into its weights. Independence has to be constructed.

It never touches production. The judge is a development instrument, not a runtime guardrail. It grades output during the dev cycle to find where the writer's prompts leak; it adds zero latency and zero cost to serving. Grading one diamond costs about thirty cents — four passes, each re-reading the full evidence — which is trivially cheap in development and pointlessly expensive in production, where the deterministic gates already stand. A judge is not a content filter. It is how you find out what to fix.

Part III — Calibrate the instrument

The uncomfortable part: the judge is also a language model. It is an unreliable narrator hired to audit an unreliable narrator. This is less absurd than it sounds — auditors are people too — but only if the workflow plans for it. Ours does, in three ways.

The judge can be wrong, and the loop says so. Our written dev cycle is: the judge finds a violation → we fix the writer's prompt, or we fix the judge if the judge is wrong → we log it → we re-run. The same humility applies to the deterministic gates. We ban the word flaw — the marks inside a stone are characteristics, never flaws. Then the assistant, explaining a modest luster rating, said it was "not a flaw; just part of why the rating sits at four instead of five or six" — and the regex failed the most honest sentence in the script. So the ban was narrowed: the corrective "not a flaw" now passes, while "no flaws in this stone" — a perfection claim — still fails. The exemption got its own tests. False positives are not noise; they are calibration data.

The verdict is schema-checked, and malformed verdicts retry. In real runs the judge would intermittently drop a required field, or return a field with the wrong type. We do not average around that; the harness detects the malformed verdict and retries it, and every consumer of the verdict is guarded against a bad shape. An instrument with unhandled failure modes doesn't stop measuring — it starts lying quietly.

One sample is noise. The same stone, judged twice, does not score the same. We watched one diamond score fifteen violations on one run and six to eight on reruns. If we had compared two models using single samples, we could have concluded anything at all. The discipline is boring: average at least two runs before believing any comparison, and never let a single verdict decide anything important.

An instrument you never calibrate is just another opinion.

Part IV — Three arguments we never had to have

Here is what actually changed when the judge existed. Three decisions that AI teams normally settle by taste, seniority, or vendor reputation got settled by reading the same number before and after.

Which model. We ran two writer tiers over the same stones, with the same prompts and the same judge. Claude Haiku 4.5 — the small, fast tier — averaged about ten real violations per diamond. Claude Sonnet 5 averaged 0.8. A twelve-fold honesty gap, on identical inputs. And the direction of the tradeoff surprised us: Sonnet also scored higher on specificity, so the honesty did not cost richness. Every arithmetic blunder in the corpus came from the small model — including the run where it declared one stone "59 carats more" than another. The real gap between those stones was 0.13 carats — and fifty-nine dollars. The model had turned a price difference into a weight. Fifty-nine carats is not an engagement ring; it is a museum piece. A model that cannot keep dollars and carats apart is not a bargain at any latency.

The judge also caught something no benchmark would have: the fast tier existed only to shave seconds off time-to-first-word, yet it had quietly generated every record in our persisted "quality" collection. The latency shortcut had poisoned the quality artifact. No unit test flagged it, because every unit test passed. The judge flagged it, because the judge reads output.

The ending is the good part. The fast model's one advantage evaporated when we fixed a parser to accept the larger model's output format — and measured end to end, the strong model at low reasoning effort was fast enough to serve. Reputation said "the small model is fine for this." The instrument said otherwise, and the instrument had receipts.

Whether RAG. Everyone's default architecture for grounding is retrieval. We shipped something smaller: a deterministic key-join. The gemological knowledge — vetted definitions, approved analogies, the boundaries of what a report can and cannot tell you — is keyed by characteristic, and each stone's generation receives only the blocks matching what its report actually lists: a cloud, a feather, a needle — the trade's names for the tiny marks inside a stone. An exact lookup, not a search. So when the assistant explains a cloud, it uses the vetted definition — a cluster of microscopic pinpoints — and keeps the vetted boundary: the report cannot tell you whether it causes visible haze in this stone. The knowledge base is content-hashed into the cache key, so editing one entry regenerates exactly the output that depended on it.

Did we lose anything by skipping the vector database? That is normally a religious question. For us it was an empirical one: the judge held the grounded output at zero real violations per stone. Retrieval remained unnecessary for a bounded, one-stone-at-a-time task — not as an article of faith, but as a measured result we can revisit the day the task stops being bounded.

There was a quiet confirmation along the way. A gemological standard written independently by the domain's own experts — people who had never read our prompts — was later diffed against our system. It landed on the same rules: the same evidence discipline, the same refusal to infer eye-cleanness from a clarity grade, the same banned reassurances. Two instruments, built apart, giving the same reading. That is what being right feels like from the inside: unremarkable.

Did the edit help. The prompts are the product, so we treat them like one. Every prompt edit carries a changelog entry with four fields: the case (the concrete finding that forced it — almost always a judge finding), the reason (the principle it serves), the change (what actually moved), and the verification (a judged real run, with numbers). A typical case: the judge caught the assistant volunteering why a stone had earned its value rating — a cause the evidence never states — so the rule against manufactured explanations was widened, and the verification showed the pattern gone across redraws with the honesty counts unchanged. About sixty entries accumulated in the first week. No silent edits, ever. And because the prompt version is folded into the cache key, no cached episode can ever have come from an unknown prompt: edit the prompt, and everything it touched regenerates by design.

This is test-driven development for a non-deterministic system. The judge's verdict is the failing test you drive to green. And the tests that matter run against the real API, on real output — because in this project, every finding above came from a real run, and the mocked tests passed the whole time. Over four hundred automated tests guard what can be asserted deterministically; not one of them counts as evidence that the writing got better. Only the instrument says that.

Part V — What generalizes

Diamonds are the setting, not the point. This pattern transfers to any team shipping generative output where being wrong is expensive. The short version:

1. Build the instrument before polishing the product. You cannot improve what you argue about. A judge you trust converts taste into experiments.
2. Layer the gates, and decide which gate owns which rule. Prevent in code what can be prevented. Hard-fail on regex what is unambiguous. Send to the judge only what requires reading. The boundary is a design decision — make it with test evidence.
3. Demand a triage, not a score. Typed questions, verbatim quotes, named evidence, cleared claims reported alongside violations. If you cannot re-check a verdict, you do not have an instrument; you have a mood.
4. Keep channels orthogonal. Honesty, provenance, tone, formatting — measured separately, never blended. A single number is where problems hide behind virtues.
5. Construct independence. A different model, ideally a different lab, reading from the same written standard as the writer. Never let a model grade its own homework.
6. Calibrate, and distrust single samples. Fix the judge when the judge is wrong, guard against malformed verdicts, average runs. Variance is real and it will lie to you.
7. Keep the judge out of production. It is a development instrument. The gates that serve customers should be deterministic, cheap, and boring.

The payoff is cultural as much as technical. Model selection stopped being a debate about vendor reputation. Architecture stopped being a debate about fashion. Prompt engineering stopped being a vibe. We stopped arguing about which model was smarter and started reading the same number twice.

One honest caveat, because our instrument would flag us if we skipped it: the judge measures whether the assistant is honest about a diamond. It cannot measure whether a shopper wants to listen. That is a different experiment, with a different instrument — and no model gets to grade it for us.