Four statistical problems that get education papers rejected

You have the manuscript finished, a full school year of fieldwork behind you and a decision that is not about writing: whether to submit as is, or to open the analysis file one more time. The hesitation is reasonable, because redoing the numbers costs weeks and in education almost nobody tells you what is wrong until the letter arrives. What follows are the four problems that take an education manuscript down in the first round, and in three of the four the reviewer is not arguing with your result: they are saying that the number you computed is not the one that answers your question.

Four hundred and eighty pupils are almost never four hundred and eighty observations

The error that sinks the most education papers is neither exotic nor sophisticated: it is treating pupils as if each one had arrived at your study on their own. They do not arrive that way. They arrive in classrooms, and classrooms arrive in schools. Two pupils in the same class share a teacher, a timetable, classmates, materials, the time of day the test was administered and very often the neighborhood. They resemble each other more than either resembles a pupil in another school, and that resemblance is not noise: it is structure, and your Student's t does not know it exists.

The amount of resemblance has a name and can be measured. The intraclass correlation (ICC) is the share of total variance explained by differences between classrooms or between schools, and in education it is neither esoteric nor small. The What Works Clearinghouse, the US Department of Education body that reviews evidence on educational interventions, adopts default values of ICC = .20 for achievement outcomes and .10 for everything else when a study does not report one, drawing on Hedges and Hedberg (2007) and Schochet (2008). That is not a desk figure either: using the longitudinal data systems of seven US states, school-level intraclass correlations in grades 3 to 8 average around .20 in mathematics and .17 in reading, with considerable variation across states (Hedges & Hedberg, 2013). Checked in September 2026.

What that figure does to your analysis takes one line to compute. The design effect is deff = 1 + (m − 1) × ICC, where m is the average classroom size. Take 20 classrooms of 24 pupils (480 in total, a sample that sounds comfortable in education) and an ICC of .20: deff = 1 + 23 × .20 = 5.6. Your effective sample is not 480 people, it is 480 / 5.6, roughly 86. And because the standard error is understated by the square root of the design effect, yours is too small by a factor of 2.37: that t = 3.10 you obtained with p < .01 is, computed properly, t = 1.31.

This is not a result getting slightly weaker. It is a result changing its conclusion, and the reviewer who sees it is not arguing about your intervention: they are saying the contrast you ran is not the contrast that answers your question. That is why this error rejects more than any other in the discipline. It does not get fixed with a sentence in the Discussion.

How it shows up in the letter, and what to do about it

The comment almost never says «multilevel». It says «the unit of assignment does not match the unit of analysis», or «the authors should account for the nested structure of the data», or, in its shortest and deadliest form, «please justify the assumption of independence». All three are the same sentence. If your intervention was assigned by classroom and your analysis compares pupils, that sentence is already written into your next letter.

There are three ways to answer it, and the choice depends on how many classrooms you have, not how many pupils. A multilevel model. A random intercept for classroom (and one for school if you have enough schools) puts the variance where it belongs and returns honest standard errors; it is the default option and the one the reviewer expects to see. Cluster-robust standard errors. You keep your usual regression and correct only the covariance matrix, clustering by classroom; it is easier to explain and enough when the classroom is a nuisance rather than an object of study. Moving the analysis up to the classroom. You compute each classroom's mean and compare 20 observations instead of 480; it throws information away and many people find it painful, but it is honest, easy to defend and sometimes the only thing your number of classrooms allows.

Here comes the uncomfortable part. With four or six clusters in total, neither multilevel models nor robust errors behave well: variance components are estimated badly and cluster-robust errors come out optimistic, so you need small-cluster corrections and you need to say so in the Method. And the practical consequence is the hardest one to accept: if you assigned by classroom, your power comes from the number of classrooms. Adding 200 more pupils to the same six classes buys you almost nothing; a seventh and an eighth class does. That calculation belongs before the study and in the paper afterwards, and how to report it is in how to report a power analysis in APA 7.

The intervention with nothing to compare it against

The second problem is not about computation, it is about design, which is why it hurts more: one group, a pretest, a posttest and a rise. It rises, yes. The question is relative to what. Between September and May your pupils have matured, have seen the test once and have had nine months of ordinary schooling with or without your program, and not one of those three factors carries your name.

That this is not a grumpy reviewer's pet peeve is stated by the most formal body in the field. The What Works Clearinghouse does not even review single group pretest-posttest designs: it declares them ineligible because they confound time with study condition and, in its own words, are unable to provide trustworthy evidence of effectiveness; if it did make them eligible, it writes, they would always receive a rating of not meeting its standards (Procedures and Standards Handbook, version 5.0, checked in September 2026). An education journal does not apply that handbook, but its reviewers have read the same argument.

If you are still in time, the comparison group does not have to be a randomized trial. A parallel class in the same year group and the same school will do; a waiting list that gets the program in the second term will do; a cohort from the previous year with the same test on the same dates will do. What will not do is nothing, and what does worst of all is nothing in disguise: writing «the program improved reading comprehension» with a single group is the sentence the editor underlines. Once you do have two groups, the effect size for that design is not the usual d either, it has its own (Morris's d_ppc2), and that is worked through with the arithmetic in how to measure whether an educational intervention really worked.

And if the data are already collected and no control group is possible, there is one way out that is not a trick: relabel the work. A pilot or feasibility study, with the gains described, no causal language anywhere and a limitations section that says exactly what you cannot conclude, gets published. What does not get published, or gets published and then miscited for ten years, is the same study claiming the program works.

Likert: the problem is not the decision, it is the silence

In education almost everything interesting is measured with ordered response scales: attitude toward mathematics, classroom climate, engagement, satisfaction with the subject, self-efficacy. And almost everybody computes means with them. The reviewer does not mind that you do it; they mind that you do not say so. The sentence that gets you rejected is not «we treated the items as continuous», it is that this sentence appears nowhere.

Before deciding, separate two things that get confused constantly. A single five-option item is an ordinal variable with five categories and not much more to discuss. The score of a twelve-item scale, summed or averaged, is another matter: it has many more possible values and behaves rather well as continuous. When you model the items directly, the evidence is more concrete: Rhemtulla, Brosseau-Liard and Savalei (2012) compared robust maximum likelihood with categorical least squares estimation in confirmatory factor analysis, and factor loadings and their standard errors came out more accurate under the categorical method, especially below five categories; factor correlations and model fit were recovered equally well by both, and maximum likelihood was the one most sensitive to asymmetric thresholds.

That is where this field's specific trap sits. School climate and attitude scales are very often built with four options and no midpoint, precisely to force a choice, and four is below the line. Add that the anchors are almost never equidistant (the distance from «never» to «sometimes» is not the distance from «often» to «always», however you code them 1, 2, 3 and 4) and you have both of the comments the reviewer is going to write.

What closes the matter is three sentences in the Method: how many categories the scale has, whether you treat it as ordinal or continuous, and why. If your design sits on the border, add a fourth: that you ran the analysis both ways and the conclusions do not change, or that they do change, which is also a result and is better coming from you. And if you are going to report means, do not leave them alone to two decimals: 3.42 versus 3.61 means nothing without the spread and without an effect size beside it.

The missing effect size, in the field where small differences do matter

The fourth is the easiest to fix and the one that leaves the most manuscripts without an argument. The APA reporting standards ask, for each inferential test you conduct, for the effect-size estimate and its confidence interval whenever possible; it is literally in JARS–Quant Table 1. A Results section carrying only p does not breach an abstract rule: it leaves the reader without the one number that says whether the program is worth it.

And here the discipline carries a problem of its own, because it has inherited benchmarks that are not its own. Cohen's thresholds (0.20 small, 0.50 medium, 0.80 large) come, in Kraft's own description, from a handful of tightly controlled social psychology lab experiments from the 1960s performed largely on undergraduates. Kraft (2020) recomputed the scale from the distribution of 1,942 effect sizes drawn from 747 randomized trials of pre-K to grade 12 education interventions with standardized achievement test outcomes, and proposes different cut-offs for that class of study: below 0.05 is small, 0.05 to less than 0.20 is medium, and 0.20 or greater is large. The median of that distribution is 0.10.

Translated into your manuscript: if you get 0.15 and write «a small effect by Cohen's standards», you are throwing your own result in the bin. In field-based educational interventions, 0.15 sits above the median of what the entire discipline achieves. The honest move is not to inflate it or shrink it, it is to say what you are comparing it against. And it works the other way too: a d of 0.80 out of three classrooms, with no confidence interval and no clustering correction, impresses nobody who knows how to read it.

Because the effect size from a classroom-level trial is not the usual one either. When the unit of assignment is the classroom and the standard deviation in the denominator is the pupil-level one, you need a clustering correction that depends on the ICC and the number of classrooms; the What Works Clearinghouse handbook builds it into its own computation formulas, drawing on Hedges (2007). If your d carries neither that correction nor a confidence interval, you have problem one and problem four packed inside the same number. The reporting conventions, interval included, are in how to report Cohen's d in APA 7.

All four arrive together, and there is an order

These four problems present themselves as a list, but in a real manuscript they are hooked together. Ignoring the nesting inflates t and narrows the effect size's interval at the same time. Having no comparison group leaves that effect size meaningless, however perfectly it was computed. And deciding whether the scale is ordinal or continuous changes the standard deviation that goes in the denominator. Fixing them out of order means computing the effect size three times.

The order that saves work is this one. First, the unit. What was assigned (pupil, classroom, school) and what you are going to analyze; if they do not match, decide now between a multilevel model, robust errors or classroom means. Second, the comparison. What the change is measured against, and if there is nothing to measure it against, relabel the study before writing the Discussion. Third, the measure. Ordinal or continuous, with its three-sentence justification. And only then, the effect size, once, with its interval and with whatever correction it needs.

Before you submit, ask yourself the four questions in writing and look for the answers in the manuscript, not in your head: where do I say how many classrooms and how many schools there are, where do I say which group I compare against, where do I say how I treat the scale, where is the effect size with its interval. If any answer is not written in the text, the reviewer will ask for it, and asking costs them a minute and you three months. If you want to see the level of detail at which this gets flagged, the Previum sample report is published in full, with p values recomputed from the degrees of freedom and references checked one by one.

Several of these problems never reach a reviewer: they end in a desk rejection, with the editor returning the education paper in days. Before you submit, run a desk-reject check on your manuscript.

Frequently asked questions

My intervention ran in two classrooms, one treatment and one control. Is it salvageable?

Statistically salvageable in full, no: with two clusters, the classroom effect and the program effect are the same thing and no model separates them. What is salvageable is how you write it up. Analyze at pupil level if you want, but say explicitly that classroom is confounded with condition, strip out the causal language and present it as a pilot. Reviewers forgive a small design that declares itself; they do not forgive a small design sold as a trial.

Can I put classroom in as fixed-effect dummies and skip the multilevel model?

Sometimes yes, and it is a legitimate route when classrooms are a nuisance rather than an object of study. But watch one thing: if the intervention was assigned to whole classrooms, the classroom fixed effect absorbs exactly the variation you want to estimate and the model has nowhere to get the program effect from. Classroom fixed effects work when assignment happened within classrooms, some pupils in and some out. If it happened by classroom, you need a random intercept, cluster-robust errors or a classroom-level analysis.

I have pupils in classrooms and classrooms in schools. Do I need a three-level model?

That depends on how many schools you have, not how many pupils. With two or three schools there is no information to estimate the variance at that level: the component will sit at zero or the model will fail to converge, and you are better off entering school as a fixed effect (one or two dummies) and staying at two levels. With fifteen or twenty schools the third level earns its place. Either way, say in the Method which one you chose and why; reviewers accept almost any reasoned decision and no unreasoned one.

My Likert scale has five points and I sum twelve items. Do I need categorical estimation?

For the total score, almost certainly not: twelve five-point items give a wide range and the sum behaves well as continuous. The question is different if you model the items one by one, in a confirmatory factor analysis for instance; there, five categories sits right on the border and the recommendation depends on whether the thresholds are symmetric. In either case, write it down: the sentence that saves you the comment takes two lines.

The reviewer says my effect is small and that this makes it less interesting. Is it worth arguing?

Yes, if you argue with data rather than adjectives. The useful reply is not «small does not mean irrelevant», which reads as an excuse; it is placing your number in the distribution of your own field, saying what the program costs and how many pupils it reaches. An effect of 0.12 from a program that costs two hours of teacher training and reaches an entire year group is a different piece of news from a 0.12 in a one-to-one intervention of thirty sessions. And that argument belongs in the Discussion, not improvised in the response letter.

What if correcting for nesting leaves my result non-significant?

Then you know something you did not know before, and you know it before the editor does. You have three honest routes and hiding it is not one of them: report the effect with its confidence interval and talk about magnitude rather than a threshold, add classrooms if the calendar allows it, or present the work as what it is, a study underpowered at the cluster level whose interval does not rule out a useful effect. Publishing the corrected non-significant result is repairable; publishing the incorrect significant one is not.

Keep the order: the unit first, then the comparison, then the measure, and the effect size last, once. The last three are fixed by writing; the first one is almost always fixed by changing the model, which is why it is worth looking at before you call the manuscript finished. If you want a quick pass over what you have already written, with p values recomputed from your own degrees of freedom, Previum runs that self-assessment on your manuscript before you submit it.