How to Measure Whether an Educational Intervention Really Worked

You pilot a new reading-comprehension program in third grade. In September, your pupils average 52% on the comprehension test; in May, 68%. Sixteen points up, and someone in the staff room is already saying the program "clearly works." But before it goes into the innovation report —or the results chapter of your dissertation—, it's worth asking an uncomfortable question: how much of those 16 points is the program, and how much is simply that nine months of school happened?

Your group's gain doesn't say what it seems to say

A pretest-posttest design with no control group has a problem at its root: it doesn't separate the effect of your intervention from everything else that also changes between pre and post. Your pupils mature, they get somewhat familiar with the test format (the second time someone takes a similar reading-comprehension test, they tend to do a bit better even without having learned anything new about reading comprehension), and above all: they've had nine months of ordinary schooling, with or without your program. Any one of those three factors, on its own, can explain a non-trivial share of a 16-point rise.

The way to isolate your program's effect is to have a group that resembles yours in every way —same grade, same school if possible, same test, same dates— except that it doesn't receive the intervention. If that control group also rises, say, 6 points just from the school year passing, the effect attributable to your program isn't the raw 16 points: it's the difference between the two gains, about 10 points. That's the number that actually describes what your program did, and it is systematically smaller than the intervention group's raw gain —sometimes much smaller—.

Why standard Cohen's d isn't built for this

Once you have your two groups and their two time points, the temptation is to compute a Cohen's d and move on. The problem is that classic Cohen's d was designed to compare two independent groups at a single point in time, or two repeated measures within a single group. Your design has four means (pre and post, intervention and control) and, if you have the same pupils' scores at both time points, a correlation between pre and post that the standard formula has nowhere to put. Using the single-group d, or dividing the net gain by a standard deviation chosen without a clear rule, gives a number whose sampling variance isn't the one the formula assumes, and a confidence interval that doesn't mean what it claims to mean.

There are three common ways to get this wrong. The first is comparing raw gains without standardizing anything: "my group gained 10 points more than the control" says nothing about whether that is a lot or a little without knowing the test's scale. The second is computing d on the posttest alone, ignoring the pretest: if your intervention group already started somewhat ahead —by chance, by how the groups were formed, whatever the reason—, your program gets credit for a head start it did not create. The third, more subtle, is taking the classic two-independent-groups d formula and applying it directly to the change scores: it looks reasonable, but it ignores that the variance of a difference depends on the correlation between the two repeated measures, and that correlation is nowhere in the classic formula.

That's why a family of indices exists built specifically for pretest-posttest-control designs. The one this site uses —and the one you'll see if you ask anyone in education methodology for a "properly computed" effect size for this design— is Morris's (2008) d_ppc2: it standardizes the net gain (your group's gain minus the control's) by the pooled PRETEST standard deviation, the only one of the four the intervention hasn't yet had a chance to alter, and applies Hedges's (1981) small-sample bias correction. Morris built this estimator within a broader effort on how to combine effect sizes from repeated-measures and independent-groups designs within the same meta-analysis (Morris & DeShon, 2002): the goal was for a d from a pre-post-control trial to be comparable, without a hidden catch, to the d from a posttest-only trial.

d_ppc2 = c_P × [(intervention group gain) − (control group gain)] / pooled pretest SD

A real example of how this changes the picture: two fifth-grade classrooms (30 pupils each) trial a fractions program with manipulatives. The classroom that gets the program moves from a mean of 50 to 58 over eight weeks; the one that continues with the textbook, from 50 to 52. The net gain attributable to the program is 6 points, not 8. With a pretest standard deviation of 10 in both classrooms, d_ppc2 comes out to 0.59, with a 95% confidence interval of [0.07, 1.11]: the interval excludes zero, so there is evidence the program works, but with only 30 pupils per classroom the estimate is imprecise —the true effect could range from very small to very large—. If you want these numbers, with their confidence interval, computed on your own data without building the formula by hand, the pre-post effect size calculator with a control group does exactly this calculation and hands you back a results paragraph already written in APA 7.

You have your d. Now how do you explain it to a staff meeting?

Here comes the second problem, and it's nearly as important as the first: once you have your d_ppc2 = 0.59, or whatever value came out, what do you do with it? A thesis committee might be satisfied with the bare number. A staff meeting, a school leadership team, or a family asking whether the program is worth it will hear nothing in "d = 0.35," and walking them through the formula won't help either.

The first important nuance is that Cohen's classic thresholds (0.2 small, 0.5 medium, 0.8 large) hold up poorly in education. Kraft (2020) reviewed hundreds of rigorous educational trials and found that real effects are far more modest than those thresholds suggest: in well-run randomized trials, half the effects don't reach d = 0.10. That's why he proposes a field-specific yardstick —below 0.05, small; 0.05 to 0.20, medium; above 0.20, large—, also weighing the intervention's cost and scalability. By Cohen's yardstick, your fractions program (d = 0.59) would already be "large"; by Kraft's, it is even more clearly so, and that nuance matters when you compare your result against what's been published in the field.

The second nuance is that even with the right yardstick, a number in standard deviations still means nothing to someone who doesn't do statistics for a living. This is where translating the effect size into interpretable units comes in. There are several, and each answers a different question:

• Months of progress: an equivalence table published by the UK's Education Endowment Foundation (2018), which assigns each band of d a number of months of progress within a school year. It's the one a leadership team understands best, because it speaks their language —"months" instead of "standard deviations."
• Cohen's U3 (Cohen, 1988): the proportion of the control group that falls below the average pupil in the intervention group. It answers "how many of my control-group pupils would the average intervention pupil outperform?"
• Probability of superiority, or CLES (McGraw & Wong, 1992): the probability that, if you pick one pupil at random from each group, the intervention pupil scores higher. It answers a more conservative question than U3, because it compares two random pupils, not the average pupil against the whole distribution.

With a real example of a third-grade reading tutoring program, evaluated against a control group, that obtained d = 0.35 in reading comprehension: U3 is .637, so the average tutored pupil would outperform 64 out of 100 control-group pupils —against 50 at baseline, with no effect at all—. The probability of superiority is .598: pick two pupils at random, one from each group, and the tutored one wins a little under 60% of the time. And according to the EEF (2018) table, that same d = 0.35 equals about 4 additional months of progress over the year.

One honest caveat: months of progress are a communication convention, not a physical measure of learning. The equivalence isn't linear —what a preschooler gains in a year isn't what a secondary pupil gains— and it's debated in the literature. Use it for the staff meeting or the innovation report, always alongside the d and its confidence interval; don't let it replace the d in a paper or a thesis. With your own figures, or with the example above, the months-of-progress calculator runs all three conversions at once —months, U3 and probability of superiority, plus percentiles— and draws the two distributions so you can see at a glance how much they overlap.

Before you get here: did you have the power to detect it?

Everything above assumes your study had the capacity to detect the effect if it existed. If your intervention was assigned to whole classrooms —the norm in education, because you can't put half a class in one condition and the other half in another without contamination—, the sample-size calculation isn't as simple as counting pupils: those who share a classroom aren't independent observations of each other, and that changes how many classrooms you truly need. If you're still at the design stage, it's worth settling that question before collecting a single data point.

Frequently asked questions

Why isn't it enough to compare my pupils' average score before and after?

Because that difference mixes your program's effect with everything else that happens between pre and post: normal pupil maturation, the practice effect of taking a similar test format twice, and simply having had months of class. Without a control group going through the same thing minus the intervention, there is no way to separate how much of the rise is yours.

Can I use standard Cohen's d if I have a control group and two measurements?

Not the standard two-independent-groups formula, nor the single-group pre-post one: neither is built for the four means in this design, and applying them gives a poorly calculated confidence interval. Use an index built for pretest-posttest-control designs, such as Morris's (2008) dppc2.

What does it mean if my intervention has an effect size of d = 0.35?

By itself, not much: it's a standardized difference that needs translating. Converted into something interpretable, a d = 0.35 roughly means the average pupil in your intervention group outperforms 64% of the control group, and about 4 additional months of progress under the Education Endowment Foundation (2018) criterion.

Do Cohen's thresholds (small, medium, large) apply in education?

Poorly. Kraft (2020) shows that in rigorous educational trials, half of real effects don't reach d = 0.10, well below what Cohen would call "small." By that yardstick almost no real educational intervention would "work"; by the thresholds Kraft proposes for the field, an effect of 0.20 is already large.

Do I need to know how to code to calculate any of this?

No. dppc2 with its confidence interval, and the translation into months of progress, percentiles and probability of superiority, are available as free calculators on this site: you only need each group's means, standard deviations and sample sizes.

That gives you the two pieces you needed: the right effect size for a pretest-posttest-control design, and a way to explain it that doesn't sound like jargon. If your program is evaluated classroom by classroom, it's worth making sure you have enough power to detect the effect you expect before you collect a single data point.