Someone has to check the checker
By now you've got an LLM judge scoring your outputs on every commit, and you've learned to distrust it just enough — position bias, verbosity bias, self-preference bias, all the ways a fast, cheap measurement instrument can be systematically wrong. But distrust only gets you so far. At some point you need to know, not guess, whether the judge is actually right. And the only way to know that is to compare it against something you trust more.
That something is a human. Not because humans are infallible — trained annotators disagree with each other plenty — but because a careful human reading a response with real domain judgment is the closest thing you have to ground truth. The judge you built in the last lesson needs exactly this: a baseline it can be checked against, periodically, to confirm the numbers it produces still mean what you think they mean.
Human evaluation is also expensive, slow, and doesn't scale to gating every pull request. So the real question isn't "human or judge" — it's when the cost of a human is worth paying, and how to spend that budget so it actually produces a signal you can trust.
Vague instructions produce noisy labels
Hand two careful, competent annotators the instruction "rate the quality of this response from 1 to 5" and give them the same output to score. There's a good chance they'll disagree — not because one of them is careless, but because "quality" was never defined. One annotator weighs completeness heavily; another weighs concision. One treats a slightly-too-formal tone as a minor deduction; another treats it as a major one. The instruction left the line-drawing up to each person individually, and two people draw the line in different places.
The fix isn't better annotators. It's better instructions. A usable annotation guideline is written and example-rich: it defines exactly what "good" means for this specific task, and it works through the edge cases in advance rather than leaving them to be improvised in the moment. If a response is technically correct but buries the answer in three paragraphs of preamble, does that count as a 4 or a 2? A vague guideline never says. A good one has already answered that question before the first annotator sees a single example.
This matters because annotator disagreement is almost always diagnosed as a task-difficulty problem when it's really a guideline problem. The instinct when two labels don't match is to assume the task is inherently subjective, or that you need more experienced people. Usually neither is true — the rubric just hasn't been tightened enough to remove the ambiguity that's causing the split.
The catch: Writing a genuinely example-rich guideline takes real effort upfront — you have to go find the edge cases, not just wait for them to surface in the data. Teams that skip this step and jump straight to labeling almost always end up rewriting the guideline halfway through the pass, which means re-labeling everything scored before the rewrite.
Measure before you scale
Writing a guideline you believe is clear is not the same as confirming it actually is. The way you confirm it is by measuring inter-annotator agreement: have two or more annotators independently label the same shared subset, then measure how often they agree, corrected for the agreement you'd expect by chance alone. For two annotators, that's Cohen's kappa; for three or more, it's Fleiss' kappa.
The reason to do this on a small subset first, rather than trusting the guideline and running the full labeling pass, comes down to when you find out you were wrong. If you check agreement on fifty examples and it's low, you've lost an afternoon — you go back, find where the guideline was ambiguous, tighten it, and rerun the check. If you skip that step and only discover the disagreement after ten thousand labels have been collected, you've lost ten thousand labels' worth of signal, because you can't trust any of them to mean what you thought they meant.
Kappa gives you a number, but the real value is in what a low score forces you to do: go read the specific examples where annotators split, and ask why. Almost every time, the answer traces back to a case the guideline never anticipated — which is exactly the kind of gap that a written, example-rich guideline is supposed to close before scaling begins.
The gotcha: A high kappa score doesn't prove your guideline is measuring the right thing — it only proves your annotators agree with each other. Two people can be in perfect, well-calibrated agreement while both applying a rubric that's quietly wrong for the task. Agreement is a necessary check, not a sufficient one.
Spend the budget where the judge can't go
Given that human evaluation is slow and expensive, the practical question is where it earns its cost. Three cases stand out. The first is domain expertise the judge model itself doesn't have — legal reasoning, medical claims, deeply specialized technical content — where an LLM judge isn't just biased, it may genuinely lack the knowledge to tell a correct answer from a confidently wrong one. The second is high-stakes launch decisions, where the cost of a wrong go or no-go call is large enough that you want a human in the loop regardless of how well-calibrated your judge normally is. The third is the one that ties back to the previous lesson directly: building the initial ground-truth set you use to validate that an LLM judge is trustworthy in the first place. You can't calibrate a judge against nothing — you need a human-labeled baseline to check it against.
That third case points at the actual division of labor that production systems settle into. Human eval doesn't replace the judge, and it isn't meant to run continuously — it's too slow and too costly to gate every commit the way an automated judge can. Instead, humans calibrate and validate periodically: you pull a fresh sample, have it labeled by trained annotators using your guideline, and compare those labels against what the judge scored on the same sample. If they agree, you've got evidence the judge is still measuring what you think it's measuring. If they diverge, that's a signal to revisit the judge's prompt or rubric.
Think of it as instrument calibration. A judge that's never been checked against a human baseline is a precise-looking number with no guarantee it means anything — you're trusting a sensor you've never tested against reality. A periodic human calibration pass is what turns "the judge says 4.2 out of 5" into a number you can actually act on.
The tradeoff: Because human eval only runs periodically, there's a lag between when a judge drifts and when you'd catch it. A prompt change or a model swap on the system under test can silently degrade judge accuracy for weeks before the next calibration pass surfaces it — which is an argument for calibrating more often around any change that touches the judge's inputs, not just on a fixed calendar schedule.
Humans calibrate, judges run
The thread running through this lesson is that human evaluation and LLM-as-judge aren't competing options — they're two instruments operating at different frequencies. The judge runs constantly, cheaply, on every change, and it's good enough for that because it's been checked. Humans run rarely, expensively, and their job isn't to replace the judge's daily work but to confirm it's still trustworthy — which is only possible if the guidelines behind their labels are specific enough to produce reproducible scores, and if you've measured agreement to confirm they actually are.
Get that calibration relationship right and you get the best of both: the speed of an automated judge with the credibility of human judgment standing behind it. Get it wrong — skip the guideline work, skip the agreement check, or let calibration lapse — and you end up with a judge whose numbers look precise but mean nothing in particular.
In the next lesson, we'll take this a step further: turning ad-hoc human and judge spot-checks into a versioned, repeatable regression suite — golden datasets you can run the same way every time — and survey the framework landscape you'd actually run it with.