Who judges the judge?
Last lesson ended with rubrics — decomposing "good answer" into named, scorable criteria so a 3 out of 5 tells you something more specific than "meh." But a rubric is just a scoring sheet. Something still has to hold it up against the output and fill it in. When there's no single correct answer to check against — no reference string, no exact match, nothing deterministic — the dominant approach since 2023 has been to hand that scoring sheet to another LLM and call it the judge.
This works well enough that LLM-as-judge is now the backbone of most modern eval pipelines. It's fast, it's cheap relative to human annotation, and it scales to the volume of outputs a real system produces. But there's a catch that's easy to gloss over: the judge is itself a language model. It doesn't stand outside the system it's evaluating, immune to the quirks of the thing it's made of. It inherits LLM failure modes — and now those failure modes are operating on the act of judging itself.
Three biases show up consistently across judge models, regardless of which provider trained them: a preference for whichever answer happens to sit in a particular slot, a preference for longer answers regardless of whether the extra length says anything, and a preference for outputs that sound like the judge's own family of models. None of them make the judge useless. But none of them are things you can afford to not know about.
The slot decides before the content does
Set up a pairwise comparison — response A next to response B, judge picks the better one — and you'd expect the decision to track quality. Swap the two responses into the opposite slots and ask again, and you'd expect the same verdict. You don't reliably get it.
Judges systematically favor whichever response occupies a particular position in the prompt, most commonly whichever one is shown second. This has nothing to do with the content of either response. It's a property of where the text sits relative to the judge's attention, and it shows up as measured swings of 10 to 15 points of win rate purely from slot order — enough to flip a close comparison, or to make a mediocre variant look like a clear winner just because you happened to list it last.
The good news is that this is the easiest of the three biases to neutralize, because it lives entirely in how you frame the prompt, not in anything the judge model learned during training. Run every pairwise comparison twice, once with the responses in each order, and average the two scores. Whatever slot advantage existed cancels out across the pair.
That fix is nearly free — one extra API call per comparison — which makes it worth doing by default rather than treating it as optional hardening. It also sets a useful contrast with the next bias, which doesn't go away nearly as easily.
The catch: Position bias is measured at 10–15 points of win-rate swing from slot order alone. The fix — swap and average — is cheap precisely because the bias is a framing artifact. Skip the swap and you're not measuring quality, you're measuring list order.
Longer reads as better, even when it isn't
Ask a judge to compare a tight, three-sentence answer against a rambling, three-paragraph one that says the same thing with more words, and the judge will lean toward the longer one. Not always, but often enough to matter, and the effect holds even when the added length contributes nothing — no new facts, no better structure, just more words.
This one is structurally different from position bias, and that difference is the whole point. Position bias comes from how you frame a single prompt; you can swap slots and it disappears. Verbosity bias is baked into the judge model's training — it's a learned association between length and quality that shows up regardless of how you frame the comparison. Measured across GPT-4-, Claude-, and PaLM-2-class judges, the inflation runs 15 to 30 points of preference for the longer response, purely from length.
Because it's learned rather than a framing artifact, you can't fix it with a single clever prompt trick the way you can with position bias. Length-normalizing the score, or explicitly instructing the rubric to penalize padding, brings it down — but it doesn't erase it. The more honest way to think about it is as a persistent tax on your scores: something to account for every time you compare outputs of different lengths, not a bug you patch once and move past.
That "learned versus framing" distinction is the thread to hold onto, because the next bias sits even further on the "learned" end of the spectrum — it's not about the response at all, it's about who wrote it.
The catch: Verbosity bias measures 15–30 points of inflated preference for longer outputs, and it's baked into training, not prompting. Length-normalization and anti-padding rubric language reduce it — treat the residual as a standing tax, not a solved problem.
The judge likes its own voice
Here's the one that should give you pause if you're evaluating a system built on the same model family as your judge: the judge rates outputs from its own family more favorably than a blind human panel would rate the same outputs. It's not comparing against some external standard of quality — it's comparing against its own sense of what a good answer sounds like, and its own outputs sound the most like that by construction.
The confirmed effect size lands in the 10 to 25 percent range. That's large enough to make an evaluation pipeline structurally biased toward whichever vendor built the judge — which is a serious problem if you're using an LLM-as-judge setup to decide, for instance, whether to switch providers, or to claim your system beats a competitor's.
The mitigation is architectural rather than a prompt tweak: use a judge from a different model family than the system under test. If that's not available — say you're locked into one provider for cost or compliance reasons — ensemble judges across multiple families and require majority agreement. A single model's self-preference gets diluted when it has to out-vote judges that don't share its bias.
Position bias, verbosity bias, self-preference bias — three distinct failure modes, three distinct fixes. What they share is that none of them are reasons to throw out LLM-as-judge. They're reasons to treat it like an instrument you calibrate, which is exactly what comes next.
The catch: Self-preference bias runs 10–25% in confirmed effect size. If your judge and your system share a model family, your eval pipeline is structurally tilted toward "pass" before a single output is scored.
Trust, but check the instrument
None of the three biases above mean LLM-as-judge is untrustworthy. They mean it's a measurement instrument with known, measurable systematic error — the same way a thermometer has a calibration curve, or a scale needs to be zeroed. You don't throw out the thermometer because it reads two degrees high; you learn the offset and correct for it, or you check it against a known-good reference periodically to make sure the offset hasn't drifted.
The known-good reference here is a small human-labeled sample. Periodically — not once at launch and never again — run your judge against that sample and compare its scores to the human labels. If they track closely, your judge is calibrated and you can keep trusting its verdicts on the much larger volume of outputs no human has time to read.
If judge scores and human scores diverge, resist the instinct to go fix the system under evaluation. The divergence is telling you something about the judge, not about the thing being judged. The fix lives in the judge prompt or the rubric — tightening the criteria, adding examples of edge cases, adjusting for one of the three biases above — not in the system whose outputs you were trying to measure in the first place.
Treating calibration as a one-time setup step rather than an ongoing check is the single most common way teams end up trusting a judge that's quietly drifted out of alignment with what a human would actually say.
The catch: If judge and human scores diverge on the calibration sample, the fix is the judge prompt or rubric — not the system under test. Confusing the two means you'll "fix" a system that was never broken while the actual measurement error persists.
A calibrated instrument, not an oracle
The throughline across all three biases is the same: an LLM judge is not a neutral arbiter sitting outside the system, handing down ground truth. It's a model with its own learned preferences, its own sensitivities to framing, and its own blind spot for outputs that sound like itself. Position bias you swap away almost for free. Verbosity bias you normalize and then keep taxing your expectations against. Self-preference bias you address by diversifying which models sit in the judge's seat.
What ties all three fixes together is the same discipline: treat the judge as an instrument that needs periodic calibration against a trustworthy reference, not as a black box you trust by default because it returns a clean-looking number. A judge that agrees with your human sample today can drift as prompts change, as the system under test changes, or as the judge model itself gets silently upgraded by its provider.
That human-labeled sample you calibrate against is itself only as good as the process that produced it — and that process has its own failure modes, separate from anything an LLM judge runs into. In the next lesson, we'll turn to human evaluation workflows: when LLM-as-judge isn't enough and you need real annotators in the loop, and how to calibrate the judge against them so the two stay in agreement over time.