How to Report the ICC and Inter-Rater Agreement in APA 7

Three coders watched the same 20 therapy sessions, you opened Analyze > Scale > Reliability Analysis in SPSS, ticked the intraclass correlation box, and the output handed you two numbers — single measures and average measures — under a heading you picked from a dropdown without being entirely sure what it claimed. That dropdown is the report. A reviewer reading ICC = .87 with no model named cannot tell whether you are claiming that those three coders agree with each other or that any three coders drawn from the same population would, and those are different claims with different consequences for your study. Here is the whole line, from the three decisions that produce it to the sentence you paste into the Method. If what you need first is the coefficient itself — what it measures and how to compute it in SPSS or R — that lives in the guide to the ICC and inter-rater reliability.

What a complete ICC line contains

Six pieces, and the first three are words rather than numbers: the model (one-way random, two-way random, two-way mixed), the form (single rater or the mean of k raters), the definition (absolute agreement or consistency), the coefficient, its confidence interval and the F test with its degrees of freedom. The full template reads: “a two-way random-effects, single-rater, absolute-agreement ICC = .87, 95% CI [.72, .94], F(19, 38) = 21.08, p < .001”.

The abbreviation ICC is set upright, not italicized. The APA 7 rule is that single-letter statistical symbols are italicized — M, F, r, p — while multi-letter abbreviations stay roman, exactly as CFI, RMSEA and AIC do. The coefficient carries two decimals and no leading zero, because its ceiling is 1: .87, never 0.87, and the interval bounds follow suit. If it comes out negative the sign goes inside everything: ICC = −.04, 95% CI [−.31, .24].

There are two shorthand notations and they do not say the same thing. Shrout and Fleiss (1979) number the model and the form, ICC(2,1) or ICC(3,k); McGraw and Wong (1996) encode the definition with letters, ICC(A,1) for absolute agreement and ICC(C,1) for consistency. The practical problem is that the Shrout and Fleiss numbering welds the model to the definition — their 2 is random with absolute agreement, their 3 is mixed with consistency — so it has no way of writing a two-way mixed, absolute-agreement ICC, which is exactly what many studies need. The cheap fix is to write it out in words the first time and put the shorthand in brackets: nobody gets lost, and you are not relying on your reader remembering a table from 1979.

The F test carries n − 1 degrees of freedom in the numerator and (n − 1)(k − 1) in the denominator, with n rated cases and k raters: 20 sessions and 3 coders give F(19, 38). That p tests whether the ICC is zero, a null hypothesis nobody cares about — if your raters agreed not at all, you would have known before opening SPSS — so it is reported for completeness and the informative work is done by the interval. If you would rather not build the syntax, the ICC calculator returns the coefficient with its interval for whichever model you choose.

Three questions, and only one leaves the number alone

First, the model. One-way random when each case is rated by a different set of judges drawn from a large pool — any two residents per record, not always the same pair. Two-way random when the same judges rate every case and you want to generalize to the population they came from: “any coder trained this way would land here”. Two-way mixed when the same judges rate everything and they are the only ones that matter, because they are the coders on your study and it ends with them. One question settles the dropdown: am I talking about these raters, or about raters like these?

Second, the form. Single measures estimates the reliability of one rater's score; average measures, that of the mean of the k raters. The rule is to report whichever matches the score you go on to analyze. If your dataset stores the mean of the three coders, report the average; if 80% of the sessions were scored by one person and the others coded only to check agreement, report the single. The difference is not cosmetic: with a single-measures ICC of .87 and three raters, the average rises to .95 by plain Spearman-Brown, and reporting .95 for a score that in fact came from one person inflates the reliability of your variable.

Third, the definition. Absolute agreement asks whether the raters give the same value; consistency asks whether they rank cases the same way even if one scores everyone higher. Suppose coder C is systematically 1.2 points above the other two on the working alliance scale: consistency holds at .89, because the ordering of sessions is nearly identical, while absolute agreement collapses to .68, because the values do not match. Absolute agreement is the choice when the number itself means something — a clinical cut-off, a normed score, a comparison against another study — and consistency when all you will use is relative position.

Of the three decisions, the model is the only one that can leave the number untouched: between two-way random and two-way mixed the point estimate is identical and what changes is the claim you are entitled to make, which is why it has to be stated even though SPSS prints the same value. The form and the definition do move the result, as you have just seen. One curious consequence of this architecture is that a two-way mixed, consistency, average-measures ICC is exactly Cronbach's alpha computed on the same data, with the raters playing the part of the items; if that identity is useful to you, how to report Cronbach's alpha in APA 7 is the other face of the same calculation.

The interval is the result; the ICC is only its center

With 20 cases and three raters, an ICC of .87 arrives with an interval of [.72, .94]. That is not decorative precision: it is the difference between “moderate” and “excellent” under any of the usual thresholds, decided by data you do not have. Koo and Li (2016) recommend at least 30 heterogeneous cases and at least three raters for exactly this reason; below that, the interval opens so wide that the study cannot separate a good instrument from a mediocre one. With 10 cases and two raters, an ICC of .76 comes with an interval of [.30, .94], and there the honest sentence is that you do not know.

Koo and Li's own recommendation is to grade reliability by the lower bound of the interval rather than by the point estimate. It is a conservative convention and it makes sense: the lower bound is the worst state of affairs still compatible with your data at a reasonable level of confidence. Written out, it goes like this: “agreement between coders was good, although the confidence interval extends into the moderate range, ICC = .87, 95% CI [.72, .94]”. It costs one subordinate clause and saves you the comment from the reviewer who read that interval before reaching your adjective.

Something about the ICC is counter-intuitive and worth declaring: it is not a property of your raters, but of your raters on that sample. Between-case variance sits in the denominator, so if you code 20 sessions that closely resemble one another, the coefficient drops without anyone having got worse. The same pair of coders who reach .87 on a sample with the full range of alliance can reach .60 on a subsample of uniformly good sessions. The report gains a lot from the mean and standard deviation of the coded scores (M = 5.10, SD = 1.84): with no spread, the ICC has nowhere to come from.

And yes, an ICC can come out negative. It happens when the estimated between-case variance is smaller than the error, which is what a homogeneous sample plus poor agreement produces. It does not become zero by decree and it is not dropped from the table: it is reported as it stands, with its interval, and explained in a sentence. An ICC of −.04 says something true about your coding procedure; deleting it from the manuscript and keeping only the raw percentage agreement does not.

Koo and Li's thresholds are a convention, and not the only one

The four bands almost everybody cites come from a guideline for clinical reliability research: below .50, poor; .50 to .75, moderate; .75 to .90, good; above .90, excellent (Koo & Li, 2016). They are an agreement among researchers, not a property of the mathematics, and that is how they should be presented: “following the criteria of Koo and Li (2016), agreement was good”. The citation is not ornament, it is what turns an adjective into a traceable decision.

You can see they are conventions by putting them beside Cicchetti's (1994), also widely cited and cutting elsewhere: below .40, poor; .40 to .59, fair; .60 to .74, good; .75 to 1.00, excellent. An ICC of .72 is moderate for Koo and Li and good for Cicchetti, on the same data, with nobody lying. Hence the practical rule: choose the benchmark before you see the number, cite it, and do not swap benchmarks halfway through the Discussion because the other one sounds better.

The threshold that really governs is the one your use imposes. An ICC of .75 can be adequate for a variable entering a between-groups comparison, where measurement error is spread around and mostly attenuates, and clearly inadequate for a decision about one individual, where that same error becomes an interval around their score that may straddle the cut-off. If your coding decides who enters a trial or who meets criteria, the ICC's obligatory companion is the standard error of measurement in the units of the scale, which is the quantity a clinician can actually interpret.

One writing detail turns up in review more often than you would expect: the benchmark's adjectives describe agreement, not validity. Three coders can agree excellently on a badly operationalized variable, and the ICC will be high because they agree about the same thing, the same poor definition included. High agreement is necessary before you believe the measure and it is not sufficient, so the Discussion should not write the ICC as though it certified that you measure what you say you measure.

When it is kappa and not the ICC

The data type decides, not taste. If your raters produce a number with interpretable distances — an alliance scale from 0 to 10, minutes of behavior, a quality score — the ICC applies. If what they do is sort cases into categories — meets or does not meet criteria for major depression, which of four diagnoses, present or absent — then Cohen's kappa applies, because an ICC computed on codes 1, 2, 3 and 4 would treat a mix-up between 1 and 4 as worse than one between 1 and 2, and in a nominal variable that means nothing. Two clinicians classifying 120 interviews agreed on 88.3% of cases, κ = .71, 95% CI [.57, .85], p < .001. The Cohen's kappa calculator returns the index with its interval, and the exact sentence and its traps are in how to report Cohen's kappa in APA 7.

In between sits ordinal data, which is where half of clinical research lives: severity in four levels, stages, improvement scales. There you want weighted kappa, which penalizes distant disagreements more heavily than neighboring cells, and you have to say which weights, because they change the number: with linear weights a two-category discrepancy counts twice a one-category one, with quadratic weights four times. In a typical four-level severity example, unweighted kappa comes out at .52 and quadratically weighted kappa at .78, and both are correct while describing different things. The bridge back to the previous section is exact: quadratically weighted kappa is asymptotically equivalent to the absolute-agreement ICC computed on those same categories (Fleiss & Cohen, 1973).

Cohen's kappa is for two raters. With three or more and nominal data you want Fleiss' kappa (1971), and if on top of that you have empty cells, raters who did not score every case, or you want one index covering nominal, ordinal and interval levels, Krippendorff's alpha handles all three at once. What does not work in any of the three scenarios is reporting raw percentage agreement alone: it does not subtract what chance would have produced, and 88% agreement on a category that shows up in 6% of cases sits below the 88.7% that chance alone already predicts, which is to say a negative kappa.

The serious warning with kappa is the prevalence paradox (Feinstein & Cicchetti, 1990). If two reviewers classify 200 records and only 12 are flagged as incidents, they can agree on 92% of cases and still return κ = .29, 95% CI [−.04, .62], an interval that crosses zero. It is not that the reviewers misunderstand each other: with a category that rare, chance agreement is already 88.7% and kappa has almost no room left. When it happens, report percentage agreement and kappa, and show the table margins. A sibling error lives on the continuous side: Pearson correlation does not measure agreement, because a rater who scores everyone two points high returns r = 1.00 with dismal absolute agreement; that is what the absolute-agreement ICC and the Bland-Altman plot are for.

The whole paragraph written out, and the five errors

Here it is in the Method, with nothing left implicit: “Three trained psychologists independently coded working alliance in the 20 sessions (M = 5.10, SD = 1.84). Inter-rater reliability was estimated with a two-way random-effects, single-rater, absolute-agreement intraclass correlation, ICC(2,1) = .87, 95% CI [.72, .94], F(19, 38) = 21.08, p < .001, which under the criteria of Koo and Li (2016) corresponds to good agreement, with a lower bound in the moderate range. Subsequent analyses used the mean of the three coders, whose reliability was ICC(2,3) = .95, 95% CI [.89, .98].” Five lines, and the reader knows which model, on which sample, with what precision, and which score entered the analyses.

The categorical version is just as short: “Two independent clinicians classified the 120 interviews as meeting or not meeting criteria for a major depressive episode. Raw agreement was 88.3% and Cohen's kappa was κ = .71, 95% CI [.57, .85], p < .001, substantial agreement under Landis and Koch (1977). The 14 discrepancies were resolved by consensus with a third clinician blind to condition.” That last item — what happened to the disagreements — is missing from most manuscripts and is the first thing a methodological reviewer asks about, because it determines whether your final variable is a coding or a negotiation.

Five reasons this analysis comes back, in descending order of frequency. One. A bare “ICC = .87”, with no model, form or definition: the number without the question it answers. Two. Average measures reported when the score you analyze was produced by a single coder, which is borrowed reliability. Three. Consistency where absolute agreement was needed, typical when the scores are later compared against a cut-off. Four. No confidence interval, so nobody can tell whether 20 cases or 200 sit behind it. Five. The adjective pinned to the point estimate while the interval's lower bound sits two bands below it.

A sixth is rarer and more expensive: reporting the ICC of a subset while letting the reader assume it covers all the material. If double coding was done on a random 20% of sessions, say so, with its n, because the interval depends on that number and not on the study total. And if the subset was not random — the first ones, or the ones that looked hard — then the resulting ICC is not the reliability of your procedure, and that sentence belongs in the manuscript too.

Already drafted it? Paste your Results section into Review one section: it reads it the way a reviewer would and returns the first objection, attached to the sentence that causes it.

Frequently asked questions

How do you report an ICC in APA 7 format?

By naming the model, the form and the definition first, and then the number with its interval: “two-way random effects, single rater, absolute agreement, ICC = .87, 95% CI [.72, .94], F(19, 38) = 21.08, p < .001”. The abbreviation ICC is set upright rather than italicized, because it is a multi-letter abbreviation rather than a single-letter symbol. Neither the coefficient nor the interval bounds take a leading zero, since an ICC cannot exceed 1. The line belongs in the Method, alongside the number of coded cases and the number of raters.

Which ICC model should I choose, two-way random or two-way mixed?

It depends on whether you are talking about your specific raters or about raters like yours. Two-way mixed is for when the study's judges are the only ones that matter and you are not generalizing; two-way random treats them as a sample from a population of similarly trained coders, which is what you need if you argue the procedure is replicable in another team. The point estimate SPSS prints is the same either way: what changes is the claim, which is why it must be declared. One-way random is a different number and applies only when each case was rated by different judges.

What is a good ICC value?

The most cited convention is Koo and Li's (2016): below .50 poor, .50 to .75 moderate, .75 to .90 good and above .90 excellent, judged on the lower bound of the confidence interval rather than the point estimate. It is an agreement among researchers, not a law: Cicchetti's (1994) thresholds cut elsewhere and call .72 good where Koo and Li call it moderate. Pick one before you see your result, cite it, and remember that decisions about individual people demand more reliability than comparisons between groups.

What is the difference between single measures and average measures ICC?

Single measures estimates the reliability of one rater's score; average measures, that of the mean of the k raters who took part. Because averaging cancels error, the second is always the larger: .87 for single measures with three raters becomes .95 for the average. Report whichever matches the score that goes into your analyses. If your dataset stores the mean of three coders, report the average; if each case carries one person's score and the others coded only a subset to check agreement, report the single.

When do you use kappa instead of the ICC?

When raters sort cases into categories rather than scoring quantities. For two raters and nominal categories, Cohen's kappa; for ordered categories, weighted kappa with the weighting scheme declared as linear or quadratic; for three or more raters, Fleiss' kappa or Krippendorff's alpha, which also tolerates missing data. The ICC is reserved for continuous variables and numeric scales with interpretable distances. The borderline case is a many-point ordinal scale, where treating it as continuous is common practice and quadratically weighted kappa returns essentially the same number as the absolute-agreement ICC.

Why is my ICC low when my raters agree?

Almost always for one of two reasons. The first is a homogeneous sample: between-case variance sits in the ICC's denominator, so if every session you coded resembles the others, the coefficient falls even when the judges agree to the decimal. The second is that you asked for absolute agreement and there is a systematic bias, with one rater scoring everybody higher; consistency comes out high and absolute agreement collapses. Before you redo the training, look at the standard deviation of the scores and compare each rater's mean: the diagnosis is right there.

With the model, the form and the definition named, the coefficient without its leading zero, the confidence interval and a cited benchmark, your ICC is reported the way APA 7 asks; and if your raters are assigning categories, the same discipline applies to kappa with its weights declared. When the hard question is not the format but whether your coding design supports the conclusion you want from it — how many double-coded cases, what to do with disagreements, whether the agreement is good enough to decide about individuals — that is what my statistical consulting is for.