Power Analysis for Mixed Models and GLMs: Where G*Power Stops

The ethics committee wants a sample size justification, and what you have is a cluster-randomized trial across fourteen clinical teams, measurements at baseline, three and six months, a binary outcome and four covariates. You open G*Power and look for the menu entry that says «mixed model with a random slope and a binary outcome». It is not there. What is there is a t test, so you enter d = 0.50, get 128, and that number goes into the protocol. The trouble is not that it is optimistic. It answers a question about a design that is not yours, and the gap between the two numbers runs into hundreds of participants.

Where G*Power stops, and why that is not a flaw

G*Power is neither broken nor obsolete. Its menu of test families —exact tests, F, t, chi-square and z— describes contrasts in which every observation contributes the same amount of information, that information is independent of its neighbor, and all the uncertainty sits in a single error term. Under those conditions power has a closed-form solution: you evaluate the relevant noncentral distribution and get an exact answer in a second. A large share of experimental psychology lives there, and for that work nothing beats it. That territory is covered in the guide to calculating sample size with G*Power.

The two modules that look like they might rescue you sit under «z tests»: logistic regression and Poisson regression. They exist and they work, but the conditions are tight: a single predictor, the marginal distribution of that predictor declared by hand (normal, binomial, uniform, exponential and so on), every other covariate compressed into one «R² other X» field, and a Wald approximation. They take no interactions, no exposure offset, no overdispersion and —above all— no clustering of the observations. They cover the textbook case, and your design almost never is one.

The pattern repeats what you may already have met in sample size for mediation: the question you are asking corresponds to no distribution the software has tabulated. There the culprit was the ab product; here it is random effects and the mean-variance link built into generalized models. The way out is the same one, and it is older than G*Power: simulate.

The issue is not the effect, it is how much each person tells you

When participants are nested inside units, two people from the same clinic resemble each other more than two people from different clinics, and that resemblance has a number: the intraclass correlation coefficient (ICC). The design effect turns it into lost sample with a one-line formula, DEFF = 1 + (m − 1) × ICC, where m is the cluster size. A burnout intervention with twenty staff per team and an ICC of .05 carries a design effect of 1.95: your 400 staff hold the information of 205 independent people. G*Power counts 400. The distance between those two figures is your study.

That produces the finding that reshapes most protocols: recruiting more people inside each unit hits a ceiling. The effective number a cluster contributes is m / (1 + (m − 1) × ICC), so with an ICC of .05 a cluster of twenty is worth 10.3, a cluster of fifty is worth 14.5, and an infinitely large cluster is worth 20, which is 1 / ICC. At that ICC you will never extract more than twenty effective participants from a single unit, however many you enroll inside it. Power is bought with units, and units are the expensive part. Before running any numbers it helps to be clear on how the model is built, which is what the multilevel models guide covers.

In figures: if the t test asked for 128 participants (64 per group, d = 0.50, α = .05 two-tailed, power .80), that design effect pushes you to 250, which in whole units and balanced arms means fourteen clusters of twenty (280), seven per arm. And seven per arm opens the second problem, which is no longer about size but about degrees of freedom. With few clusters the naive Wald test rejects more often than its stated α, so you need Kenward-Roger or Satterthwaite degrees of freedom, which are less powerful than the naive version. McNeish and Stapleton (2016) reviewed where it starts to bite: below roughly thirty to forty clusters the correction stops being optional.

None of this is an implementation detail you can leave until the analysis. The simulation has to apply the same test with the same correction you will report, because corrected and uncorrected power are not the same quantity. And when treatment is allocated to the unit, the sample that matters is counted in units: Maas and Hox (2005) showed that below about fifty groups the standard errors of the variance components come out biased downwards.

In repeated measures, the random slope eats your power

The repeated-measures ANOVA module in G*Power asks for one correlation among the measurements and an epsilon correction. The model underneath is compound symmetry: the same correlation between any pair of occasions and the same variance at each. Epsilon does not change that model, it only adjusts the degrees of freedom of the F test when the assumption fails; what still stands is that the data have to be complete and balanced, and anyone who misses a session drops out of the analysis entirely. It is precisely the set of assumptions that sent you to a mixed model in the first place, as the comparison of mixed models and repeated-measures ANOVA sets out. Powering under those assumptions and analyzing without them means planning one study and running a different one.

The expensive part is random slopes. If people differ in how much they change over time, or in how strongly they respond to the manipulation, the within-person effect is tested against that between-person variability in slopes and not against residual noise alone. Power drops, and it drops in a way you cannot fix by measuring the same people more often: past a point, each extra measurement sharpens that individual's slope and adds almost nothing about the average slope. Powering with random intercepts and then fitting random slopes, which is what the maximal structure of Barr et al. (2013) asks for, is the fastest route to an underpowered study that looked comfortable on paper. Matuschek et al. (2017) argued the trade-off between maximal structure and power; what neither position licenses is settling it after seeing the data.

With crossed random effects the problem doubles. In a study with participants and items, the items are a second sample and they have a size too: thirty participants with eight stimuli is not thirty participants with forty, even though the number of people is identical. Brysbaert and Stevens (2018) worked that calculation through in detail and, for the designs they examine, end up recommending on the order of 1,600 observations per condition. Kumle, Võ and Draschkow (2021) published the equivalent tutorial with R code and the mixedpower package.

Attrition enters through a different door. A mixed model uses all available information under a missing-at-random assumption, so someone who drops out at month three still contributes their first two measurements instead of vanishing as they would from an ANOVA. The penalty is real but smaller, and it depends on when people leave rather than only on how many leave. That is not something to estimate in your head: you simulate it, with the dropout rate and its timetable written into the generating model.

In a GLM the effect size does not travel alone

In linear regression the error variance is a free parameter: you can talk about d = 0.50 without ever saying what the control mean is. In a generalized linear model you cannot. The variance is a function of the mean —p(1 − p) for a binary outcome, equal to the mean for a Poisson— so the same coefficient carries different power depending on where the baseline rate sits.

The numbers make it concrete. To detect an OR of 1.5 with α = .05 two-tailed and power .80: if the control rate is 50 %, the treated group sits at 60 % and you need about 385 per group. If the control rate is 5 %, the treated group sits at 7.3 % and you need about 1,685 per group, more than four times as many people for the same odds ratio. Any justification that quotes an odds ratio without stating a baseline rate is incomplete, and in clinical and health services research the baseline is usually low: relapse, readmission within 90 days, self-harm episodes, treatment dropout.

Counts behave the same way. If you power under a Poisson assumption and then analyze with a negative binomial because your episode counts are overdispersed —and counts of self-harm, substance use or crisis contacts almost always are— the real standard errors are larger than the ones you simulated and the promised power never materializes. The dispersion parameter is one more input to declare, alongside zero inflation and an exposure offset when people are observed for different lengths of time.

There is a useful side effect here. Because no clean d exists for a coefficient in a generalized mixed model, the input has to be stated on the model's own scale: points on the questionnaire, an odds ratio, a rate ratio. That forces you to say out loud what change you consider worth detecting before you collect anything, which is the conversation nobody has and the one a reviewer is glad to find already written down.

The simulation, step by step

Six steps, none of them hard on its own. One. Write a data-generating model with the same shape as the analysis model: same fixed effects, same random structure, same family. Two. Fix every parameter: the effect you want to detect, the standard deviations of the random effects, their correlation, the residual, and the baseline rate if the outcome is binary or a count. Three. Simulate a few thousand datasets with the design you will actually have, attrition included. Four. Fit the real model to each one and run the real test, with the same degrees-of-freedom correction or the same likelihood ratio test you intend to report. Five. Count the proportion of replications that reject: that proportion is your power. Six. Repeat across a grid of sample sizes and plot the curve, because a single point never tells you whether you are on a plateau or on a slope.

Two control figures. Simulated power is itself an estimate with sampling error: with 1,000 replications and a true power of .80 the 95 % margin is roughly ±.025, and with 5,000 it drops to ±.011. A thousand replications to explore, five thousand for the number that goes in the protocol, and always the seed in writing. The second figure is convergence: singular fits and warnings are not free. Decide in advance what you do with them and publish the rate. Counting them as non-significant is conservative and honest; discarding them and computing power only among the replications that converged conditions the answer on the model behaving well, and that inflates it.

Existing tools save you from writing this from scratch. simr (Green and MacLeod, 2016) works directly on an lme4 model and ships powerSim, powerCurve and extend for growing the design; mixedpower does the same from the Kumle et al. tutorial; the MONTECARLO command in Mplus covers multilevel and structural equation models; PowerUpR and Optimal Design give closed-form answers for the cluster-randomized designs that fit their templates; WebPower runs in the browser. Any of them beats an invented number, and all of them ask for exactly the same ingredients.

The uncomfortable question is where those ingredients come from with no prior study to lean on. The answer is not to invent a value and present it as known: it is to run a grid. Simulate across the plausible range —ICC from .02 to .08, a difference between 2 and 4 points— and report the region where the design holds up. «With an ICC as high as .08 and a difference of 3 points, fourteen units keep power above .80» is a defensible sentence. A single number pretending to a certainty you do not have collapses at the first question.

How it gets written in the Method

What separates a justification that passes from one that comes back is not the number, it is whether somebody else can reproduce it. A t test needed six pieces of information, the ones collected in how to report a power analysis in APA 7. A mixed model or a GLM needs nine.

Model. The equation with its random structure spelled out: what is random and within what. Hypothesis. Which specific parameter is tested and with which test. Effect. The smallest effect of interest on the model's own scale, and where it comes from. Variances. The standard deviations of the random effects and the residual, or the equivalent ICC, with their source. Design. Number of clusters, cluster size, measurement occasions and allocation ratio. Attrition. Expected rate, timetable and the assumed missingness mechanism. Criterion. α, its tails, and target power. Simulation. Number of replications, seed, and what was done with non-converging fits. Software. Name, version, and packages with versions.

In prose it reads like this: «Power was estimated by Monte Carlo simulation with 5,000 replications (seed 20260904) in R 4.4.1 using lme4 1.1-35.5 and simr 1.0.7. The generating model matched the analysis model: burnout score with fixed effects of group, time and their interaction, a random intercept for team, and random intercept and slope for participant. Components were fixed from the pilot study: team intercept SD = 1.8, participant intercept SD = 5.4, slope SD = 1.2, intercept-slope correlation = −.20 and residual SD = 6.1, equivalent to a team ICC of .05. The group × time interaction was tested against the model without the interaction by a 1-df likelihood ratio test at α = .05 (non-directional), with an expected difference of 3.0 points at six months. Attrition was set at 15 % by six months, missing completely at random. With fourteen teams and twenty staff per team (N = 280), estimated power was .84, 95 % CI [.83, .85]; with ten teams it fell to .71. Convergence warnings occurred in 0.7 % of replications and were counted as non-significant. The simulation code is in the project repository».

An experienced reviewer reads that paragraph looking for three things: whether the simulated model is the same as the analyzed model (the classic slip is simulating random intercepts and fitting random slopes), whether the variance components have a source or are wishful thinking, and whether the number of clusters survives the degrees-of-freedom correction the analysis will need. Once all three are on the page, the discussion moves to your design instead of your arithmetic.

Frequently asked questions

Can I use G*Power for a multilevel or mixed model?

Not for the contrast you care about. Its test families assume independent observations and a single error term, and there is no menu entry for random effects anywhere. What you can do is use it as a starting point: compute the N the same contrast would need without nesting and multiply by the design effect, 1 + (m − 1) × ICC. That is a reasonable approximation for a main effect of group, and it breaks down as soon as you have random slopes, cross-level interactions or a non-normal outcome.

Is it better to have more clusters with fewer people, or fewer clusters with more people?

More clusters, nearly always. A cluster contributes at most 1 / ICC effective participants however large it grows: with an ICC of .05, twenty people in one unit are worth 10.3, fifty are worth 14.5, and the ceiling is 20. Going from twenty to fifty per unit buys you 4.2 effective participants; opening one more unit of twenty buys you 10.3. When treatment is allocated at the unit level, the number of units also governs the degrees of freedom for the test.

How many replications does the simulation need?

A thousand to explore, five thousand for the final figure. Simulated power is a proportion and carries sampling error: with 1,000 replications and a true power of .80 the 95 % margin is roughly ±.025, and with 5,000 it drops to ±.011. Always report the number of replications and the seed, because without them nobody can reproduce your justification, and report the percentage of replications that failed to converge as well.

I have no pilot data for the ICC. What do I put in?

A range, not a value. Published compilations exist by field: in primary care and health services research ICCs usually sit below .05 (Adams et al., 2004), whereas in classrooms and academic attainment they run closer to .10 to .20 (Hedges and Hedberg, 2007). Simulate across that range and report the region where the design keeps its target power. It is more defensible than a single value, and it tells you in advance how much slack you have if the real ICC turns out worse than planned.

Does the ten-events-per-variable rule work for logistic regression?

Not as a power calculation. The rule from Peduzzi et al. (1996) concerns bias in coefficient estimation rather than the probability of detecting an effect, and van Smeden et al. showed it does not reliably do even that job. Riley et al. (2019) proposed different criteria for prediction models. None of these rules says anything about nesting or about the baseline rate, and the baseline rate is what actually decides the power of your contrast.

The data are already collected. Can I still justify the sample size?

Not with post hoc power computed on the observed effect, which is just the p value rewritten. What holds up is a sensitivity analysis, and simulation does it just as well: fix the design you ended up with —your clusters, their sizes, your real attrition— and find the smallest effect you would have detected with power of .80. That is a statement about what your design could see, and it survives review.

The short version: G*Power computes the power of a test, not the power of a model. As soon as your design carries a second source of variance or an outcome that is not normal, the calculation stops being a menu lookup and becomes a simulation you have to write, document and defend. If what is in front of you is a protocol with a deadline, a handful of units and an ICC you only half know, building that simulation and leaving the Method paragraph written is exactly the kind of job I take on through my statistical consulting.