Inter-annotator agreement measures how consistently different raters assign labels to the same items under a specified coding scheme. A quant team might check whether two analysts identify the same adverse events in earnings-call passages before using those labels to train a classifier. Disagreement can expose an ambiguous definition, inconsistent instructions or genuinely difficult cases.
Agreement is evidence about reproducibility. Accuracy requires a reference treated as correct; construct validity asks whether the label captures what the research intends to measure. Two analysts can agree on a keyword rule that misses the intended concept. The Measurement Validity lesson develops that distinction (public preview; full lesson requires membership).
Start with independent labels, not a consensus file
Define the unit first: a sentence, call turn and whole filing are different tasks. Carletta, §3, shows why the choice of unit matters for comparisons. Write the event definition and borderline examples in a codebook. Have raters label a common sample without seeing each other's decisions. Preserve these original labels before discussion or adjudication: agreement after everybody adopts one consensus label does not measure independent reproducibility. Artstein and Poesio, §4.1.1, explain why fixed instructions and independent coding matter.
Include the text population, sample selection and rater selection in the report. A collection of clear-cut event sentences answers a different question from a sample dominated by boilerplate. For human–model teams, retain the model version, prompt and settings as well as the human instructions; the LLM Annotation lesson covers this workflow (public preview; full lesson requires membership).
A 100-passage audit
Here are invented labels, not the results of an annotation experiment. Each of two annotators labels all 100 passages once. Yes (+) means the passage explicitly reports the defined adverse event; no (−) means it does not. Rows are annotator A; columns are annotator B. Neither axis is a truth reference, so the cells are not called true positives or false negatives.
| Annotator A / Annotator B | Yes (+) | No (−) | A total |
|---|---|---|---|
| Yes (+) | 40 | 10 | 50 |
| No (−) | 5 | 45 | 50 |
| B total | 45 | 55 | 100 |
The diagonal contains 85 exact matches:
A uses yes on 50% of passages; B uses it on 45%. Their no proportions are 50% and 55%. For two raters assigning nominal categories, Cohen kappa compares the observed match rate with the product-of-marginals benchmark (provided ):
The observed agreement covers 70% of the distance from this benchmark to perfect agreement. It does not say that 70%, or 85%, of labels are correct. Cohen's own retrospective explains the marginal-frequency motivation for the coefficient.
“Chance” names a hypothetical independent pairing at each rater's observed category frequencies. It does not establish that either rater guessed randomly, or that a particular match was caused by chance. Independent working procedures and the benchmark's statistical independence are different ideas.
Original synthetic samples with yes as the positive event label. Blue diagonal cells count exact agreements. The second sample has 85 agreements, all on no, and no joint positive labels; neither table contains a truth reference.
The second table changes the yes/yes count to zero and no/no to 85, leaving the two disagreement counts at 10 and 5. A now says yes on 10% of items and B on 5%. Raw agreement is still 85%, but
Agreement is slightly below this sample's benchmark. All matches are on no, and the raters identify disjoint sets of event passages. Label frequencies changed; the true event prevalence remains unknown. This illustrates why raw agreement, the full table and both marginals belong beside kappa. Artstein and Poesio, §3, discuss prevalence and marginal effects; a single verbal rating band hides these differences.
Match the coefficient to the annotation design
Inter-annotator agreement is the broad problem; Cohen kappa is one statistic. Changing the design changes what must be specified:
| Annotation design | What needs attention |
|---|---|
| Two raters, nominal categories | Ordinary Cohen kappa counts every category mismatch equally. |
| Ordered categories | Weighted kappa can distinguish a near miss from a distant disagreement. Declare category order and weights before inspecting the result. |
| More than two raters | Specify a multi-rater coefficient and its chance model; Fleiss's coefficient is not simply Cohen kappa with extra columns. |
| Incomplete ratings | A suitable Krippendorff alpha can use available within-item comparisons. State the measurement level and missing-rating pattern. |
Krippendorff's methodological note, sections C–D, explains incomplete ratings and measurement-level choices. A lone rating on an item supplies no within-item comparison. Supporting missing entries does not make selective skipping harmless: omitting the hardest passages can change the population actually assessed. These coefficients and weighting choices are not interchangeable adjustments to maximise a reported score.
Use disagreements to improve the measurement
Inspect contested passages by category and reason, revise the codebook on development material, then assess the revised procedure on fresh independently labelled items. Keep adjudicated training labels distinct from the original agreement audit. The Inter-Rater Agreement lesson connects reliability, marginal frequencies and the target population of financial texts (public preview; full lesson requires membership).
A sample coefficient has uncertainty. For population claims, report a suitable confidence interval, with an inference method that respects the sampling design; many passages from one filing are not automatically independent observations. No interval is calculated for these fictional tables. Set acceptance criteria according to the intended use and the cost of disagreements, rather than treating a conventional kappa band as a validity certificate.
Sources
- Jean Carletta (1996). Assessing Agreement on Classification Tasks: The Kappa Statistic, §§3–4 (choice of annotation unit, intended use and diagnosing disagreements). Historical terminology combines variants; the coefficients are distinguished here.
- Jacob Cohen (1986; commentary dated November 1985). Retrospective on “A coefficient of agreement for nominal scales”. Author's explanation of the marginal benchmark and corrected-agreement ratio; not the original 1960 full text.
- Ron Artstein and Massimo Poesio (2008). Inter-Coder Agreement for Computational Linguistics, §§2.4–2.6 (coefficient choices), 3 (marginal and prevalence effects), 4.1 (coding design and interpretation).
- Klaus Krippendorff (2011; literature updated 2013). Computing Krippendorff's Alpha-Reliability, introduction and sections C–D (multiple observers, missing ratings and measurement levels).