How Many Classrooms Does Your Study Need for Real Power

A doctoral student in Education designs a study to evaluate a reading-comprehension program: 4 classrooms will get the program, 4 will continue with usual teaching. With about 22 pupils per classroom, that's roughly 176 pupils in total —a sample that, calculated as if every pupil were independent, is more than enough to detect almost any reasonable effect—. The problem is that this arithmetic is wrong from the start, and not by a little: it treats 176 dependent observations as if they were 176 independent ones, and the difference between those two things is, more often than not, the difference between a study with real power and one doomed to find nothing even if the program works.

Why a classroom isn't 22 independent pupils

Two pupils in the same classroom share a teacher, a teaching pace, a classroom climate, classmates, and even the time of day they take the subject. All of that makes their results resemble each other more than they would if pupils were scattered at random across every classroom in the school. In statistical terms, they aren't independent observations: each additional pupil within the same classroom contributes less new information than a pupil from a different classroom, because part of what that pupil measures has already been contributed by their classmates.

A sample-size calculation that ignores this —the one you get from multiplying "classrooms by pupils" and treating the result as loose pupils recruited one by one— drastically underestimates how many classrooms are needed. This isn't a minor nuance: it's the difference between a study that can detect the effect it's looking for and one that, even if the program genuinely works, has no real chance of proving it.

The intraclass correlation coefficient: how alike classmates are

The intraclass correlation coefficient (ICC) is the proportion of the total variance in the outcome explained by belonging to one classroom rather than another, as opposed to differences between pupils within the same classroom. An ICC of 0 would mean a classroom contributes no resemblance at all —pupils in the same class wouldn't resemble each other any more than two random pupils in the school—, and the naive per-pupil calculation would be correct. A higher ICC means the classroom "weighs" more in the outcome, and the more it weighs, the less new information each additional pupil from that same classroom brings.

In practice, the ICC is almost never known precisely for your specific context before you start, so you fall back on published reference values. For academic-achievement outcomes, Hedges and Hedberg (2007) document, from US data, values typically between 0.10 and 0.25. For attitudinal outcomes —motivation, classroom climate, well-being—, the ICC tends to be lower, between 0.05 and 0.10. These are planning values, not a universal law: the prudent approach is to state the source of the value you use and, where possible, check how your calculation changes with somewhat higher or lower values.

The design effect: how the ICC turns pupils into classrooms

The design effect is the factor that multiplies the "naive" sample size —the one that would assume full independence— to account for the ICC and the typical size of a class. Donner and Klar (2000), in their reference work on cluster-randomized trials, put it this way:

DE = 1 + (m − 1) × ICC

where m is the number of data-contributing pupils per classroom. With ICC = 0, DE = 1: no penalty, as if pupils were independent. As the ICC or the class size grows, DE grows with it, and that factor directly multiplies how many pupils you need.

A real example shows the size of the problem. You want to evaluate a reading-comprehension program in fifth grade against usual teaching. You expect a d of 0.35 —already a large effect for education, under the Kraft (2020) thresholds we covered in how to measure whether an educational intervention really worked—, each classroom contributes about 22 pupils with consent and completed measures, and you take the medium academic-achievement ICC, 0.15 (Hedges & Hedberg, 2007). With a two-tailed test, α = .05 and 80% power:

• If you treated pupils as independent, you would need 129 pupils per arm.
• The design effect with m = 22 and ICC = 0.15 is DE = 1 + (22 − 1) × 0.15 = 4.15.
• The corrected sample size is 129 × 4.15 ≈ 536 pupils per arm.
• Divided by the 22 pupils per classroom, that's about 25 classrooms per arm —50 in total, roughly 1,100 pupils—.

The contrast that explains everything: if the ICC were 0 —classmates not resembling each other at all—, the design effect would be 1 and 129 / 22 ≈ 6 classrooms per arm would suffice. Going from 6 to 25 classrooms changed neither the expected d, nor the power, nor the pupils per classroom: only the fact that pupils come in groups that resemble each other on the inside. Ignoring the ICC doesn't make the study cheaper; it makes an underpowered study look sufficient on paper.

And there's a ceiling worth knowing: however many pupils it holds, a classroom never contributes more information than, roughly, 1/ICC independent pupils. With an ICC of 0.15, that ceiling is about 7 "effective" pupils per classroom —a class of 22 already contributes close to 5, and doubling it to 44 barely lifts that to 6—. Going from 22 to 30 pupils per class changes the answer by a classroom or two; going from 20 to 30 classrooms changes the entire study. That's why, when power is the problem, the lever that actually works is almost never enrolling more pupils in the same classrooms: it's recruiting more classrooms.

To run this calculation with your own expected d, your classroom size and the ICC that fits your outcome —with a sensitivity table showing what would happen under a somewhat different ICC, ready to paste into your Methods section—, the classrooms-needed calculator works through it link by link and leaves you the paragraph already written.

Why your study with 3 or 4 classrooms per condition is at risk

The limiting case is the one most worth understanding. With 1 or 2 classrooms per condition, no later analysis can separate the intervention's effect from that specific classroom's own effect: if classroom A improves more than classroom B, you will never know whether it was the program or that particular teacher, that group of pupils, that time slot. The intervention and the classroom are confounded by design, and no statistical model, however sophisticated, can fix afterward what the design left unresolved.

With 3 or 4 classrooms per condition —a number that sounds "reasonable" in many dissertations and school projects because it already implies several hundred pupils— the problem doesn't disappear, it only softens: as the example above shows, for a large effect in education (d = 0.35) and a typical ICC, roughly 25 classrooms per condition were needed, not 3 or 4. A study with 4 classrooms per condition and 800 pupils is still, for power purposes, a small study with many repeated measures of the same handful of groups. If you plan to analyze with a multilevel model —the correct way to analyze nested data—, that model will work with the information the design collected, not the information it wishes it had: real power is set by the number of classrooms, not the number of pupils.

You have your classrooms: what do you compute once the data come in?

Once enough classrooms are recruited and the pre and post data are collected, the question shifts from design to analysis, and the answer depends on whether you have a control group. If you do —the case covered here—, the right effect size is the one that compares the net gain between conditions, as explained in how to measure whether an educational intervention really worked. If what you want is to describe, classroom by classroom, how much each group learned of what it had left to learn —with or without a control group, as a descriptive complement—, the Hake normalized gain calculator does exactly that, with Hake's (1998) bands as a reference for whether that gain is low, medium or high.

Frequently asked questions

Why can't I calculate sample size treating each pupil as independent?

Because pupils in the same classroom share a teacher, a pace and an environment, which makes their results resemble each other. Each additional pupil from an already-recruited classroom contributes less new information than one from a different classroom. A calculation that ignores this treats dependent observations as independent and drastically underestimates how many classrooms are needed.

Where do I get the intraclass correlation coefficient for my study?

From a published, stated value, unless you have your own prior data from your schools. For academic achievement, Hedges and Hedberg (2007) document typical values between 0.10 and 0.25; for attitudinal outcomes, 0.05-0.10 is usually enough. Since you will never know it precisely, it's worth checking how the result changes under somewhat different values before you fix the number of classrooms.

What is the design effect, and why does it multiply the sample so much?

It's the factor DE = 1 + (m − 1) × ICC (Donner & Klar, 2000), with m the pupils per classroom. It converts the sample size that would hold if pupils were independent into the one you truly need given that they come grouped into classrooms. With an ICC of 0.15 and classes of 22 pupils, for example, it multiplies the sample by more than 4.

With few classrooms per condition, isn't a multilevel model enough to analyze the data?

No. The multilevel model is the correct way to analyze nested data, but it analyzes the information the design collected, not the information it wishes it had: power is set by the number of classrooms, not the number of pupils. A study with 4 classrooms per condition is still a small study even with 800 pupils.

Does adding more pupils per classroom make up for having fewer classrooms?

Very little, past a certain point. A classroom never contributes more information than, roughly, 1/ICC independent pupils: with an ICC of 0.15, that ceiling is about 7 effective pupils, so doubling the class size barely moves the needle. The lever that actually buys power is recruiting more classrooms, not enrolling more pupils in the ones you already have.

A well-thought-out education study starts by recognizing that pupils come grouped into classrooms, not by counting them as if they were independent. Settle this before you collect data, and the analysis that follows —with the effect size or the gain that fits your design— will have something real to work with.