How to Calculate Intraclass Correlation Coefficient (ICC)

Intraclass Correlation Coefficient (ICC) answers one of the most important questions in empirical research: when two or more raters, instruments, or measurement occasions produce values for the same subject, how much can you trust those measurements? ICC is not a simple correlation. It is a reliability statistic built from variance components — and choosing the wrong form, or calculating it without understanding what it measures, is one of the fastest ways to undermine a study’s credibility.

The ICC has been the standard reliability metric in clinical, psychological, educational, and social research for decades. Its defining feature is that it partitions total score variance into two components: variance that reflects genuine differences between subjects, and variance that reflects measurement error (disagreement between raters, instruments, or occasions). The proportion of total variance explained by between-subject differences is the ICC. A high ICC means the measurements successfully discriminate between subjects — meaning the instrument is reliable. A low ICC means the error variance dominates — meaning you cannot trust that the measurement reflects anything real about the subject.

What Exactly Is the Intraclass Correlation Coefficient?

The intraclass correlation coefficient is a reliability index that quantifies the degree of agreement or consistency among multiple measurements of the same subjects — where those measurements come from raters of the same “class” (meaning there is no logical way to distinguish them, unlike distinguishing a test from a retest, or variable X from variable Y). The word “intraclass” distinguishes it from the Pearson product-moment correlation, which measures agreement between two logically distinct variables.

ICC operates by modeling total score variance using ANOVA. Scores vary because subjects genuinely differ from each other (this is the signal we want), and because raters or measurement occasions produce inconsistent readings (this is the error we want to minimize). The ICC is, in its most basic form, the ratio of between-subject variance to total variance. This simple idea has remarkable consequences: it means an ICC computed in a very homogeneous subject sample will be artificially low not because the instrument is unreliable, but because there is little true between-subject variance to partition.

Why ICC Replaced Pearson Correlation for Reliability

Before ICC became standard, researchers frequently misused Pearson’s r to assess inter-rater reliability. The problem is that Pearson r is insensitive to systematic rater differences. If one radiologist consistently rates tumor size 20% higher than a colleague, their readings will have a Pearson r of 1.0 — perfect linear association — yet they fundamentally disagree. Agreement-form ICC correctly penalizes this systematic offset, making it far more appropriate whenever you need to establish that different raters or instruments can be used interchangeably.

The landmark paper establishing the ICC framework was Shrout and Fleiss (1979), published in Psychological Bulletin, which defined the six canonical forms of ICC and provided formulas based on one-way and two-way ANOVA. The most widely cited interpretation guide is Koo and Li (2016) in the Journal of Chiropractic Medicine, which provides the benchmarks (poor, moderate, good, excellent) used in virtually every reliability paper published today.

The Six ICC Forms: Shrout and Fleiss (1979) Explained

The most consequential decision in any ICC analysis is choosing the correct form. Shrout and Fleiss (1979) defined six forms of the intraclass correlation coefficient organized by two dimensions: the statistical model (one-way random, two-way random, or two-way mixed) and the unit of measurement (single rater vs. mean of k raters). Choosing incorrectly — say, using a two-way formula when a one-way design was used — produces a biased and uninterpretable result.

An additional dimension added by McGraw and Wong (1996) in Psychological Methods — who extended the taxonomy to ten forms — is the choice between agreement and consistency within each two-way model. This is functionally the most important distinction for researchers: agreement ICC asks “do the raters produce the same absolute values?”, while consistency ICC asks “do the raters rank subjects the same way?”

Model 1: One-Way Random Effects (ICC(1,1) and ICC(1,k))

The one-way random effects model applies when each subject is rated by a different set of raters randomly drawn from a larger population, and when those rater identities are not tracked. There is no way to account for rater-specific effects because different subjects were rated by different people. This is the most conservative ICC form — it produces the lowest ICC estimates — because it cannot separate rater-to-rater variability from other sources of within-subject error.

Model 2: Two-Way Random Effects (ICC(2,1) and ICC(2,k))

The two-way random effects model applies when the same set of raters rates all subjects AND both subjects and raters are considered random samples from larger populations. This is the most commonly appropriate model for inter-rater reliability studies in psychology, education, and clinical research. ICC(2,1) can be calculated in both agreement and consistency forms: the agreement form penalizes systematic mean differences between raters; the consistency form does not.

Model 3: Two-Way Mixed Effects (ICC(3,1) and ICC(3,k))

The two-way mixed effects model requires the same raters to rate all subjects but treats those specific raters as fixed effects — the only raters of interest — rather than as a random sample. This model is appropriate when the same two or three specific instruments are used throughout a study and you do not intend to generalize to other instruments. The correct rule: if you plan to generalize your reliability conclusions to other raters, use ICC(2). If limited to these specific raters, use ICC(3).

ICC Formulas: ANOVA-Based Calculation from First Principles

Calculating the intraclass correlation coefficient by hand requires running a one-way or two-way ANOVA and extracting the Mean Square components. Understanding why the formula uses these particular components reveals what the ICC actually measures and why it behaves the way it does in small samples.

Step 1: Organize Your Data

Your data matrix should have n rows (subjects or targets) and k columns (raters or measurement occasions). Every cell contains the rating assigned to subject i by rater j. For ICC to be meaningful, the same construct must be measured by all raters using the same scale.

Step 2: Run the ANOVA and Extract Mean Squares

Step 3: Apply the ICC Formula for Your Model

Step 4: A Worked Numerical Example

Consider a study where n = 6 subjects are each rated by k = 3 raters on a pain scale (0–10). The two-way ANOVA yields: MSB = 14.28, MSR = 0.48, MSE = 0.72. We compute ICC(2,1) absolute agreement:

Interpreting the ICC: Koo and Li (2016) Benchmarks

Once you have calculated the ICC, interpretation requires more than looking up a benchmark bin. The numerical value means different things depending on the ICC form used, the homogeneity of your subject sample, and whether you are reporting a point estimate or a confidence interval.

The Koo and Li (2016) Benchmarks

Agreement vs. Consistency: Which ICC to Report?

How to Select the Right ICC Form: A Decision Framework

Choosing between ICC forms is one of the most frequent sources of error in reliability research. The following decision framework walks you through the correct selection process, following Koo and Li (2016).

Decision Question 1: Are the Same Raters Used for All Subjects?

No → Use ICC(1). Yes → Proceed to Question 2.

Decision Question 2: Generalize to Other Raters, or Only These Specific Raters?

These specific raters only (fixed): Use ICC(3). Generalize to other raters (random): Use ICC(2) — most inter-rater reliability studies.

Decision Question 3: Does Systematic Rater Bias Matter in Practice?

Yes (absolute values drive decisions) → Agreement ICC. No (only rank order matters) → Consistency ICC.

Decision Question 4: Individual Rater or Mean of k Raters?

Individual rater: Use (1) form — e.g., ICC(2,1). Mean of k raters: Use (k) form — e.g., ICC(2,k).

Calculating ICC in R, SPSS, and Excel

Calculating ICC in R: The irr and psych Packages

Calculating ICC in SPSS

Navigate to Analyze → Scale → Reliability Analysis. Move rater columns into the Items box. Click Statistics → check “Intraclass Correlation Coefficient.” Select Model (Two-Way Random for ICC(2), Two-Way Mixed for ICC(3)), Type (Absolute Agreement or Consistency), and 95% Confidence Interval. Click OK. SPSS returns the ICC value, 95% CI, F-test, and degrees of freedom. Note: SPSS does not compute ICC(1) — use R for the one-way random model.

Calculating ICC in Excel (Manual ANOVA Approach)

Run Data → Data Analysis → Anova: Two-Factor Without Replication. The output table gives SSB (Row SS), SSR (Column SS), and SSE (Error SS). Compute MSB, MSR, MSE, then apply the formula directly:

Key Figures, Institutions, and Tools in ICC Research

Patrick E. Shrout and Joseph L. Fleiss — Columbia & New York University

Patrick E. Shrout (NYU) and Joseph L. Fleiss (Columbia) published “Intraclass Correlations: Uses in Assessing Rater Reliability” in Psychological Bulletin in 1979. This paper defined the six canonical ICC forms, provided ANOVA-based formulas, and established confidence interval procedures. It remains one of the most cited papers in clinical measurement — with over 30,000 citations as of 2026. The Shrout-Fleiss (1979) framework is the mandatory citation for any paper or assignment reporting an ICC.

Terry K. Koo and Mae Y. Li — New York Medical College

Koo and Li published “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research” in the Journal of Chiropractic Medicine in 2016. Their paper synthesized the literature, provided practical decision rules for model selection, and established the poor/moderate/good/excellent benchmarks. They explicitly recommended reporting the 95% CI lower bound as the reliability classification criterion — the second mandatory citation for any ICC analysis.

Kenneth O. McGraw and S.P. Wong — Texas A&M University

McGraw and Wong extended the Shrout-Fleiss framework to ten ICC forms in their 1996 Psychological Methods paper, formally introducing the agreement-versus-consistency distinction within two-way models. This is the citation for the agreement/consistency distinction in any ICC report.

The irr and psych Packages in R

The irr package provides icc() with explicit model/type/unit arguments. The psych package, by William Revelle at Northwestern University, provides ICC() returning all six Shrout-Fleiss forms simultaneously with confidence intervals. Both are freely available on CRAN and are the de facto standard for ICC analysis in academic research.

IBM SPSS Statistics

The dominant statistical platform in many clinical and social science programs. Its Reliability Analysis procedure implements ICC(2) and ICC(3) with both agreement and consistency options via a point-and-click interface — the most accessible ICC platform for students without programming backgrounds.

Where ICC Is Used: Applications Across Research Fields

Clinical and Health Research: Measurement Reliability

In clinical research, ICC validates the reliability of physical measurements — range of motion, blood pressure, pain scale ratings, radiological measurements — before those measurements are used as outcomes in trials or diagnostic criteria. The standard is a reliability substudy of n ≥ 30 patients assessed by two or more raters, with ICC(2,1) absolute agreement reported alongside its 95% CI. Organizations like GRAPPA and OARSI typically require ICC ≥ 0.85 with CI lower bound ≥ 0.70 for clinical trial endpoints.

Psychology and Education: Observational Coding

In developmental psychology and educational research, trained observers code complex behaviors from video recordings. ICC — specifically ICC(2,1) or ICC(3,1) — is the appropriate reliability index for continuous or ordinal coding. Cohen’s kappa is more appropriate for nominal coding (present/absent), while ICC handles ordered or continuous codes. In educational assessment, ICC validates essay scoring rubrics and performance-based evaluations where human judges assign numerical scores.

Multilevel Modeling: Variance Partition Coefficient

In multilevel modeling, ICC quantifies the proportion of total variance attributable to the higher-level grouping variable (school, clinic, country). An ICC of 0.15 in a school-effects model means 15% of student test score variance is between schools — a meaningful amount justifying multilevel modeling rather than standard OLS regression. The performance package in R provides icc() specifically for extracting multilevel ICC from lme4 or brms models.

Test-Retest Reliability in Questionnaire Development

When developing psychological questionnaires, test-retest reliability assesses whether the instrument produces consistent scores when administered to the same subjects 1–4 weeks apart. ICC(2,1) with the two measurement occasions treated as “raters” is the correct approach. A significant paired t-test alongside a high ICC indicates mean scores changed systematically even though relative rankings were stable — a signal of practice or learning effects.

How to Write About ICC in Assignments and Research Papers

The Complete ICC Results Statement

Every ICC result should contain six elements: the ICC form, the point estimate, the confidence interval, the sample size, the reliability classification per Koo & Li (2016), and the model justification. Example:

Essential Vocabulary for ICC

Reliability — the consistency or reproducibility of measurements. Inter-rater reliability — degree of agreement among different raters assessing the same subjects. Intrarater reliability — consistency of ratings by the same rater across occasions. Test-retest reliability — stability of measurements across time without genuine change. Agreement — whether measurements produce the same absolute values (absolute-agreement ICC). Consistency — whether measurements rank subjects in the same order (consistency ICC).

Between-subject variance (σ²ₛ) — variance reflecting genuine differences between subjects; the ICC “signal.” Within-subject variance (σ²ₑ) — variance reflecting measurement error; the ICC “noise.” Rater variance (σ²ᵣ) — systematic differences between raters; present only in two-way models. Variance partition coefficient (VPC) — alternative name for ICC in multilevel modeling; proportion of total variance explained by the clustering variable.

Spearman-Brown formula — predicts how reliability changes with number of raters: ICC(k) = k×ICC(1) / (1 + (k−1)×ICC(1)). Attenuation bias — downward bias in correlations caused by measurement error in predictors. Standard Error of Measurement (SEM) — reliability expressed in original scale units: SEM = SD_total × √(1 − ICC). Generalizability theory — a framework extending ICC to multiple simultaneous error sources, developed by Lee Cronbach and colleagues. Intracluster correlation — ICC in cluster-randomized trial design, used to calculate design effect and required sample size.

Frequently Asked Questions: Intraclass Correlation Coefficient