An LLM judge scores or compares answers under a rubric. Audit false accepts against reviewed human labels, control evaluation biases and separate agreement from truth.
LLM as a judge
LLM as a judge means using a large language model to evaluate an answer under specified criteria, by scoring it, assigning a label or comparing it with another answer. A quant team might use a judge to screen summaries of earnings calls for unsupported claims before analyst review. The judge's acceptance is a measurement to validate; it does not establish that a claim is true or that a research strategy is sound.
The evaluation object is the answer, and the judge is the evaluator. This differs from asking a model to label the sentiment of the original filing, although both need a codebook and a human audit. The LLM Annotation lesson develops that shared discipline (public preview; full lesson requires membership).
Give the judge a job it can be checked on
“Is this a good research answer?” leaves the target ambiguous. A more useful rubric separates factual support, numerical correctness, coverage of the question and clarity. For a filing summary, require that factual claims be supported by the supplied filing passages, that dates and units match, and that missing evidence be acknowledged. A fluent answer can fail the factual-support criterion.
Zheng et al., §3.1, distinguish three evaluation forms:
Form
What the evaluator receives
Meaning of its result
Single-answer grading
A question, one answer and criteria
A score or label for that answer
Pairwise comparison
A question and two answers
A preference, with an explicit tie policy
Reference-guided grading
An answer plus a reference solution
A judgement informed by that reference
A pairwise win does not imply that either answer meets an absolute acceptability standard. Nor does a numerical score become a probability merely because it lies between zero and one. Keep the rubric, supplied evidence and decision rule alongside the result.
A 40-answer audit
Consider invented reference labels and judge decisions for 40 financial-summary answers. Assume human reviewers apply a fixed rubric, discuss disagreements and adjudicate a reference label: 20 answers are acceptable and 20 unacceptable. No actual human study or model evaluation produced these counts.
Define acceptable as the positive class. In the table, rows are the human reference labels (the conventional “actual” axis); columns are the judge's predictions. “Actual” here identifies the comparison axis, not infallible truth.
Human reference label
Judge accepts
Judge rejects
Total
Acceptable
16 true positives
4 false negatives
20
Unacceptable
6 false positives
14 true negatives
20
Total
22
18
40
The resulting model evaluation summaries answer different questions:
Accuracy says 75% of all decisions match the reference. Accept precision says about 72.7% of accepted answers are acceptable under that reference. Recall says the judge finds 80% of the acceptable answers. If acceptance permits an answer to bypass further review, the six false accepts are especially relevant: the judge accepts 30% of the unacceptable answers in this toy set. That denominator is 20 unacceptable answers, whereas the share of accepted answers that are unacceptable is 6/22≈27.3%.
Synthetic 40-answer audit: rows are adjudicated human reference labels and columns are judge decisions. Acceptable row has 16 accepts and 4 rejects; unacceptable row has 6 accepts and 14 rejects. Accuracy 75%, accept precision about 72.7%, acceptable recall 80%.
Original synthetic audit, with acceptable as the positive class. The judge accepts 22 answers, including 6 unacceptable ones. These invented counts are a teaching example, not a measured model benchmark.
These proportions describe the specified set. They do not estimate a named model's performance, justify a universal acceptance threshold or transfer automatically to a different mixture of answer types. Report the sampling rule and error categories beside any real audit.
INTERACTIVE EXPLORER
Confusion matrix audit
See how false positives and false negatives change evaluation metrics.
Audit the reference and the evaluator
Human labels need their own validity argument. Reviewers may disagree because the rubric is vague, the source evidence is missing or “acceptable” combines incompatible goals. Preserve initial independent labels, adjudicate disagreements and record why the final label was assigned. Blinding reviewers to the producing model and its judge score helps keep the reference assessment independent. Agreement between a judge and people can still reflect a shared misconception. The Measurement Validity lesson explains this distinction between reliable coding and measuring the intended construct (public preview; full lesson requires membership).
Test presentation effects. Zheng et al., §§3.3–3.4, document position and verbosity effects and discuss swapping answer order. In a pairwise audit, conceal model identities, test both orders and predeclare how ties and order reversals are handled. Order consistency alone does not establish correctness.
Check affinity for generated style.Liu et al.'s G-Eval study, §4, raises concern about preference for LLM-generated summaries. This is a task-specific finding, not proof that every judge always prefers its own output. Self-enhancement is also discussed, with causal uncertainty, by Zheng et al. Audit producer-specific errors rather than assuming that a different model name guarantees independence.
For a research workflow, freeze the judge model version, prompt, rubric, references and decision rule before the final held-out assessment. Record decoding settings and repeated-run variability where relevant. Recheck after any change to those components or to the document population. Use deterministic recalculation for quantities that can be computed and source inspection for factual claims; another model's persuasive explanation does not replace either check.
“Calibrating the judge” can mean refining the rubric on development examples to improve alignment with reviewers. That is validation and tuning, followed by a fresh held-out check. Probability calibration asks a different question: among answers assigned a stated correctness probability, does the relevant event occur at that frequency? Binary accept/reject labels and a 1–5 quality score do not answer it.
For an earnings-summary screen, the defensible deliverable is a reproducible rubric, independently reviewed reference answers, error analysis and a human escalation rule. A judge score alone supplies none of those.
Sources
Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, §§3.1–3.4 (evaluation forms, biases and mitigations) and §4 (agreement evaluation). Historical experiments, not a current model ranking.
