Reliability: Cronbach's Alpha vs Omega vs ICC (Which to Report)

Reliability is a fundamental property of any psychological measure, and yet it is one of the concepts that generates the most confusion among researchers. In its most basic sense, reliability refers to the consistency or stability of an instrument's scores. A reliable instrument produces similar results when administered under similar conditions. But there are different aspects of consistency (internal, temporal, inter-rater) and different coefficients for estimating them, and choosing the correct one depends on what type of reliability you are interested in evaluating.

If what you need is agreement between raters rather than internal consistency, you can skip ahead to the ICC section below: the ICC calculator computes the variant that matches your design (random or mixed effects, absolute agreement or consistency) along with its confidence interval.

Cronbach's alpha is, by far, the most reported reliability coefficient in psychology. Its popularity is so great that many researchers use it as synonymous with reliability, without considering whether it is the most appropriate index for their data. Alpha estimates the internal consistency of a scale, that is, the degree to which the items of the scale measure the same construct. It is based on the variance of individual items and the total variance of the scale, and its value ranges from 0 to 1 (although technically it can be negative if the items are negatively correlated). Conventional cutoff points consider an alpha of 0.70 as acceptable, 0.80 as good, and 0.90 as excellent, although these thresholds depend on the context and the purpose of the measure.

Problems with Cronbach's alpha

Despite its ubiquity, Cronbach's alpha has important limitations that researchers should be aware of. The most fundamental is that alpha assumes tau-equivalence, that is, that all items contribute equally to the latent construct (they have equal factor loadings). When this assumption is violated (and it is violated in the vast majority of psychological scales), alpha can underestimate or overestimate the true reliability. In most cases, it underestimates it, which means that the true reliability of the scale is higher than what alpha suggests. But in some cases, especially when the scale is multidimensional, alpha can artificially inflate the reliability estimate.

Another problem is that alpha increases mechanically with the number of items, regardless of the quality of those items. You can have an alpha of 0.90 simply because your scale has 40 items, even though the inter-item correlations are mediocre. For this reason, some methodologists recommend also reporting the mean inter-item correlation, which is not affected by the number of items and provides a purer measure of scale homogeneity.

Omega as a superior alternative

McDonald's omega coefficient is the recommended alternative to Cronbach's alpha by most current methodologists. Unlike alpha, omega does not assume that all items have equal factor loadings, but instead estimates reliability based on a factor model where each item can contribute differently to the latent construct. This produces a more precise and generally more accurate reliability estimate than alpha. You can compute omega (and the AVE) directly from your CFA factor loadings with our McDonald's omega calculator.

There are two main versions of omega. Total omega estimates the proportion of total score variance that is due to all common factors (the general factor plus specific group factors, if any). Hierarchical omega estimates the proportion of variance that is specifically due to the general factor, excluding variance due to group factors. If your scale has subscales or dimensions (the structure you typically uncover with exploratory factor analysis), hierarchical omega tells you how much of the variability in the total score truly reflects the general construct you are supposedly measuring, versus the variability that is due to specific dimensions. This distinction is crucial for deciding whether it is appropriate to use a total score or whether you should work with the subscales separately.

The Intraclass Correlation Coefficient (ICC)

When the reliability you are interested in is not internal consistency but rather agreement between raters (interrater reliability), the appropriate index is the Intraclass Correlation Coefficient (ICC). For categorical variables, the equivalent is Cohen's kappa, which you can compute with our Cohen's kappa calculator. The ICC evaluates the extent to which different raters (judges, coders, clinicians) assign similar scores to the same subjects. A high ICC indicates that raters are consistent with each other, which is fundamental for the validity of any measure that depends on rater judgment.

The complicated aspect of the ICC is that multiple variants exist, and choosing the correct one requires considering three aspects of your study design. First, whether each subject is rated by the same raters or by different raters (random effects vs. mixed models). Second, whether you are interested in absolute agreement (that raters assign exactly the same score) or consistency (that the relative ordering of scores is the same, even if the absolute levels differ). And third, whether you will use the score from a single rater or the mean of several raters. Shrout and Fleiss (1979) described six variants of the ICC, and McGraw and Wong (1996) proposed an alternative classification system. The correct choice depends on your design and your objectives, and using the wrong variant can produce very different reliability estimates.

Practical recommendations

My general recommendation is that you always report omega instead of (or in addition to) alpha for internal consistency. In R, the psych package with its omega() function facilitates the calculation, and the MBESS package allows you to calculate confidence intervals for both coefficients. For inter-rater reliability, make sure to choose the correct variant of the ICC according to your design, and report which variant you have used (for example, ICC(2,1) for a random effects model with a single rater per measurement). The ICC() function from the psych package in R calculates all variants simultaneously, which facilitates comparison. And in any case, remember that reliability is a property of the scores in a specific sample, not of the instrument itself. A questionnaire can be reliable in one population and not in another, which is why it is important to report reliability in your own sample, rather than simply citing the reliability from the original validation study. For a complete overview of the process, see our guide on questionnaire validation in psychology.

One last thing before you submit. Reviewers rarely argue about the value of your coefficient, they argue about the justification: why you reported alpha and not omega, or which ICC variant you used and why it matches your design. If you want to read those objections while you can still act on them, run your manuscript through our free AI paper reviewer, which returns a Reviewer 2 style critique of your methods and results before a journal editor writes one.