What are the pitfalls of using an LLM as a judge?

Answer

Judges are useful where programmatic checks cannot reach, but they carry systematic biases that make naive use misleading.

Position bias. When comparing two responses, judges favour one position, often the first. Mitigate by running both orderings and keeping only results that agree, discarding the rest as ties.

Verbosity bias. Longer, more confident answers score higher regardless of correctness. This is dangerous because it rewards exactly the failure mode you most want to catch: fluent wrongness.

Self preference. Models rate their own outputs higher. Using the same model family to judge and to generate inflates scores. Use a different model where you can.

Poor calibration. Judges cluster scores in a narrow band, typically 7 to 9 on a 10 point scale, which destroys resolution. Binary or three point scales with explicit criteria work better than numeric ratings.

Criteria drift. Vague instructions such as "rate the quality" produce inconsistent results across runs. Give a rubric with concrete, checkable conditions.

The essential discipline: validate the judge against human labels before trusting it. Label a hundred examples by hand, measure agreement, and only rely on the judge where agreement is high. A judge that disagrees with you on a third of cases is generating numbers, not measurements.

Prefer programmatic checks wherever they exist. Tests passing is a fact. A judge saying the code looks good is an opinion from a system with known biases.

Common wrong answer

Reporting judge scores as though they were objective metrics, without ever validating against human judgement. It produces a number that moves and feels like progress while measuring something unknown.