ICC and Interrater Reliability: Complete Guide (SPSS and R)

When two or more raters score the same samples (observers coding behaviour, clinicians scoring structured interviews, readers evaluating essays), you need to demonstrate that the variability in scores reflects real differences between the samples and not differences between the raters. The intraclass correlation coefficient (ICC) is the standard index for this, and also for the test-retest reliability of instruments with continuous measurement scales. This guide explains when to use it, which model to choose, how to calculate it, and how to interpret the values.

ICC vs Cohen's kappa: when to use each

The ICC and Cohen's kappa measure the same thing conceptually (agreement between raters beyond chance), but for different types of data:

• ICC: for continuous or numerical rating variables (for example, a pain scale from 0 to 10, a quality-of-life score, a reaction time).
• Cohen's kappa: for nominal or categorical variables (for example, two psychologists classifying each patient as "remission / no remission"). The Cohen's kappa calculator computes the index with confidence intervals for categorical data.

The most common mistake is using Pearson correlation instead of the ICC to measure agreement: Pearson correlation captures covariation but does not detect systematic bias (if rater A always scores 2 points higher than rater B, Pearson can be 1.0 but the actual agreement is terrible). The ICC includes that bias in the calculation.

The six ICC models (and which to choose)

The ICC is not a single index: the Shrout and Fleiss (1979) framework revised by Koo and Mae (2016) distinguishes models by two decisions:

1. The statistical model:

• ICC(1): each subject is rated by different raters, selected at random from a population of raters. Cannot generalise to specific raters.
• ICC(2): all subjects are rated by the same raters, who are considered a random sample from the population of possible raters. Most commonly used in instrument validation.
• ICC(3): all subjects are rated by the same raters, who are the only raters that exist or the only ones to whom you wish to generalise. Appropriate when the instrument will always be used by those specific raters.

2. The unit of analysis:

• Single measures: reliability of the score from a single rater.
• Average measures: reliability of the average of all raters (always higher than single; only makes sense if the instrument will always be used by averaging raters).

The most common combination in questionnaire validation and clinical research is ICC(2,1): two-way mixed, single measures: same raters for all subjects, generalisation to the population of raters, reliability of a single measurement. Use ICC(3,1) only if the raters in the study are the only ones who will ever use the instrument.

How to calculate the ICC in SPSS

1. Structure your data: one row per subject, one column per rater (or per measurement occasion, for test-retest).
2. Go to Analyze > Scale > Reliability Analysis.
3. Move the columns for each rater to the Items list.
4. In the Statistics menu, tick Intraclass correlation coefficient.
5. Select the model: Two-Way Mixed for ICC(2) or ICC(3); One-Way Random for ICC(1).
6. Type: Consistency or Absolute agreement. Choose Absolute agreement for interrater reliability (detects systematic bias); Consistency only if the bias is intentional and expected.
7. SPSS reports the ICC for single measures and average measures, with 95% CI.

How to calculate the ICC in R

The irr package has the icc() function:

library(irr)
data <- data.frame(
  rater1 = c(5, 4, 3, 6, 7, 4, 5, 6),
  rater2 = c(6, 4, 4, 6, 8, 4, 5, 7)
)
icc(data, model = "twoway", type = "agreement", unit = "single")

The arguments model (oneway / twoway), type (agreement / consistency) and unit (single / average) correspond directly to the decisions in the previous section.

Reference values and interpretation

The most cited reference values are those of Koo and Mae (2016), a more recent and conservative revision than Cicchetti's original (1994) table:

Three important points before applying the table mechanically: first, always report the 95% confidence interval; with small samples (fewer than 30 subjects) the CI is so wide that the point estimate is barely informative. Second, the standard for individual clinical use usually requires ICC > .90; for group-level research, ≥ .75 is accepted. Third, the ICC model you used changes how the values are interpreted: an ICC(3,1) of .80 means something different from an ICC(2,1) of .80.

ICC for test-retest reliability

The same procedure applies to measuring the stability of an instrument over time: one column for the first administration and one for the second. Use ICC(2,1) with Absolute agreement. Define the time interval before the study (the usual for stable-trait instruments is 2 to 4 weeks). A test-retest ICC ≥ .70 is usually the minimum acceptable for research instruments; ≥ .90 for clinical or diagnostic use. COSMIN guidelines for patient-reported outcome measures (PROMs) contain the specific standards by area; the article on questionnaire validation in health sciences summarises them.

SEM and MDC: going beyond the ICC

The ICC tells you whether an instrument is reliable in relative terms (whether it discriminates between subjects), but does not tell you how much error there is in the original scale units. The Standard Error of Measurement (SEM = SD × sqrt(1 - ICC)) answers that question. The Minimal Detectable Change (MDC = 1.96 × sqrt(2) × SEM) is the threshold above which a change in a patient's score exceeds measurement noise. These two indices are essential in longitudinal clinical research and in the validation of health outcome instruments.

How to report in APA

Specify the model, the type, and the unit; report the 95% CI; and indicate the sample size: ICC(2,1) = .84, 95% CI [.71, .92], n = 40. If reporting test-retest, add the time interval. For instrument validation, the standard practice is to include a table with the ICC by dimension or item, the 95% CI and the SEM.