The sentence a reviewer is looking for is this one: κ = .82, p < .001, 95% CI [.71, .93]. Lowercase Greek kappa, not italicized, two decimals, no leading zero — it cannot exceed 1 — and with its confidence interval, which is the part almost everyone leaves out.
But before you write it there is one thing to settle, and it is what really separates a defensible report from one that falls apart in review: if your kappa is low, is it that your raters disagree, or that kappa is measuring the wrong thing? Because both happen, and you tell them apart by looking at the contingency table.
What kappa is and what it is not
Cohen's kappa (1960) measures agreement between two raters after discounting the agreement that chance alone would have produced. That subtraction is the whole point. If two people classify cases into two categories by flipping a coin, they will match 50% of the time without having agreed on anything. Raw percent agreement would say "50% agreement"; kappa says 0, which is the truth.
Two practical consequences follow:
Kappa is not a percentage and is not read as one. A κ of .60 does not mean "they match 60% of the time".
Kappa can be negative. It runs from −1 to 1. A negative value means observed agreement is lower than chance would predict, which usually points to one rater having understood the categories backwards.
What has to be reported, exactly
A complete kappa report in APA 7 carries six pieces, and four of them are not optional:
The number of units coded and the number of raters. "Two independent coders classified the 120 excerpts."
The categories and how many there are. Kappa with three categories and kappa with ten are not comparable.
Whether raters worked independently and blind. If they consulted each other, kappa is not measuring what it claims to measure, and that has to be said.
The kappa value with its 95% confidence interval. The interval has been required since the sixth edition of APA and is still missing from most manuscripts.
The raw percent agreement, alongside kappa. It does not replace kappa, but without it the paradox below cannot be detected.
What was done with disagreements. Consensus resolution, a third coder, or exclusion. This is a methodological decision and it goes in writing.
The sentence, ready to copy
"Two independent coders, blind to the hypotheses, classified the 120 excerpts into three categories (avoidance, reappraisal, suppression). Raw agreement was 91.7% and chance-corrected agreement was substantial, κ = .82, p < .001, 95% CI [.71, .93]. The 10 disagreements were resolved by consensus with a third coder."
Note that percent agreement takes one decimal and kappa two. And that the p for kappa tests whether it differs from zero, which is a rather weak null: with 120 cases, almost any kappa comes out significant. Report it because APA asks for it, but the argument is carried by the confidence interval, not the p.
Already drafted it? Paste your Results section into Review one section: it reads it the way a reviewer would and returns the first objection, attached to the sentence that causes it.
The Landis and Koch benchmarks, and why you have to cite them
The adjectives used to grade a kappa come from Landis and Koch (1977):
Below 0: no agreement. .01 to .20: slight. .21 to .40: fair. .41 to .60: moderate. .61 to .80: substantial. .81 to 1.00: almost perfect.
Two warnings about that table. The first is that the authors themselves said the cut-offs were arbitrary, chosen to give people a shared vocabulary and not derived from anything. The second is that if you use these labels, you have to cite them: writing "substantial agreement" without saying where the adjective comes from is exactly the kind of unsupported claim a reviewer flags.
Stricter alternatives exist. For instruments that will drive clinical decisions about people, .61 is not enough, and several guidelines ask for .80 as a minimum. If your kappa falls between .60 and .80 and you are validating something that will be used in practice, state explicitly which criterion you are applying and why.
The kappa paradox: high agreement, low kappa
This is the problem that sinks the most manuscripts, and it is fixable if you see it coming.
Picture two clinicians reviewing 100 records for a rare disorder. They agree on 95 cases: 93 "no" and 2 "yes". Raw agreement is 95%. And kappa comes out at .43, which on the Landis and Koch table is "moderate", barely.
It is not a calculation error. When one category is very infrequent, chance-expected agreement shoots up — almost everything will fall into "no" by base rate alone — and the subtraction eats nearly all the observed agreement. This is what Feinstein and Cicchetti (1990) described as the first paradox of kappa.
The second paradox is the opposite: when raters have opposing biases — one over-diagnoses and the other under-diagnoses — kappa can come out artificially high.
What do you do about it? Three things, and all three go in writing:
Always report the marginal frequencies, or the contingency table itself. Without it, nobody can tell whether your kappa is affected by prevalence.
Report raw agreement alongside kappa. The distance between the two numbers is what reveals the paradox.
If prevalence is extreme, consider PABAK (prevalence-adjusted bias-adjusted kappa) as a complementary measure, never as a replacement. And explain why you are adding it.
Weighted kappa, Fleiss and ICC: which one is yours
Plain Cohen's kappa treats every disagreement as equally serious, and often they are not.
If your categories are ordered (mild, moderate, severe), confusing "mild" with "severe" is worse than confusing "mild" with "moderate". That calls for weighted kappa (Cohen, 1968). And you have to say which weights: linear or quadratic. Quadratic weights penalize distant disagreements far more and are the norm in clinical scales. A weighted kappa without stated weights is a number nobody can replicate.
If you have more than two raters, Cohen's kappa does not apply: it is built for two. That calls for Fleiss's kappa (1971). And if the set of raters changes from case to case, not even that: you need a random-effects model.
If what they are coding is continuous — a score, minutes, an intensity — kappa is not the measure. That calls for the intraclass correlation coefficient, and you have to pick the right form for your design. It is worked through in ICC and inter-rater reliability.
And one frequent case that gets handled wrong: if you are measuring the internal consistency of a scale, it is none of these. That is Cronbach's alpha or McDonald's omega, which measure something else entirely.
For the calculation, the Cohen's kappa calculator returns the value, its standard error and the confidence interval from the contingency table.
Mistakes that come up again and again
Reporting kappa without a confidence interval. The most common one. A κ = .70 from 40 cases and one from 400 are very different things, and the interval is the only thing that shows it.
Calling a kappa "substantial" without citing Landis and Koch. The adjective has an owner.
Using plain kappa with ordinal categories. It throws information away and usually underestimates agreement.
Using Cohen's kappa with three or more raters. It shows, and Fleiss is what applies.
Computing kappa on the subset where agreement occurred. It sounds absurd written down, but it happens when disagreements are resolved by consensus first and kappa is computed afterwards. Kappa is computed on the original, independent codings, always.
Not saying what happened to the disagreements. Ten disagreements resolved by consensus and ten disagreements dropped from the sample are two different studies.
Kappa is not in the book "Report Your Results in APA 7", but scale reliability is (chapter 9), and so are the formatting and table rules that govern this sentence too, on a complete study with a reproducible dataset and R, SPSS and jamovi scripts.
Before you submit
Inter-rater agreement is one of those short sections a methodological reviewer goes straight to, because five lines are enough to tell whether the work was done properly. If you want to see what will be raised before you submit, upload the manuscript to the AI paper reviewer: it is free and hands the critiques back pinned to the sentence that triggers them.
References
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220.
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543-549.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.