The Effect Theory Said Would Be There (and Wasn't)

You had a named hypothesis. Three papers said daily social media use predicts depressive symptoms, you put it in the model expecting to see it, and the output hands you β = .08, p = .354. Now you are sitting in front of the Discussion picking between three bad exits: writing that your study failed to replicate, blaming the sample in a throwaway clause, or quietly dropping the predictor and pretending it was never in the model. There is a fourth exit, and it starts with a two-minute check almost nobody runs.

First check whether there is a discrepancy at all

The framing error sits in the first sentence you write: the effect did not replicate. You never tested your result against the literature. You tested it against zero, which is the only thing a p value does. Those are two different comparisons, and you have run one of them.

Run the missing one, and run it with the interval. In the worked example, β = .08 with SE = .086 gives t(123) = 0.93 and a 95% confidence interval of [-.09, .25]. That interval contains zero, which is why p = .354. It also contains the β = .22 the source study reported. Your data are compatible with the published effect and with no effect at the same time: they cannot tell the two apart. Writing that the finding failed to replicate claims something your own interval will not support, and the interval is the part of the result a reviewer actually reads (how to read a confidence interval).

The opposite case exists and it is the one that makes you work. If the source study reported β = .45 and your interval tops out at .25, the published effect is outside and you genuinely have something to explain. The four explanations below apply either way, but their job changes: with the published effect inside your interval they explain why you could not see it; with it outside they explain why it is different. Which of the two you are in decides the verb of the whole Discussion.

Explanation 1: you were never powered for the unique effect

This is the first one by a distance, and hardly anyone writes it well, because you have to start by admitting you are estimating something else. The literature almost always reports the bivariate correlation between predictor and outcome. Your model reports the unique contribution: what the predictor adds once the others have explained everything they can. If your predictor shares variance with them, those two numbers are not remotely alike.

Follow the example with the numbers in view. In your sample, daily social media use correlates with depressive symptoms exactly as the literature said: r = .30, t(124) = 3.50, p < .001. Sleep quality, the other predictor, correlates .52 with the outcome and .45 with social media use. Put both in the same regression and the social media coefficient drops to β = .08, with a semi-partial correlation of sr = .07, roughly 0.5% of unique variance, while the model as a whole reaches R² = .28. The bivariate effect has not vanished from your data. You asked the model a different question.

Then the power arithmetic, which is where the real size of the problem shows. To detect r = .30 with N = 126 you had 93% power: that test could hardly fail. For the unique contribution, with f² = 0.0075, power falls to 16%. With that design, five times out of six the result comes out exactly as yours did even if the effect is real and exactly as large as the literature says. Reaching 80% would have taken roughly 1,040 participants, more than eight times what you recruited.

What you cannot do is compute observed power from your own effect and paste it into the Discussion. Post hoc power is a monotone transformation of your p value: if p = .354, observed power will always come out low, and it adds nothing the p had not already told you (why post hoc power answers nothing). What you report is a sensitivity analysis: the smallest effect this design could have detected.

Explanation 2: you did not measure the same thing

Two papers that both claim to measure screen use can be measuring things that correlate .60 with each other. A validated problematic-use questionnaire is not a single self-reported hours-per-day estimate, and neither of them is a device log. Change the instrument, change the version, change which subscale becomes the score, change the time frame (last week against a typical week), change the informant, and the coefficient moves. None of those differences appears in the abstract you built your hypothesis on.

Reliability does the rest quietly. An observed correlation is capped by the reliability of both measures: if the true correlation is .35 and the outcome has a reliability of .90, a scale with α = .92 would show you .32, while a four-item short form with α = .66 would show you .27. That is roughly 15% of the effect gone before anything else happens, purely because the protocol had to fit inside twenty minutes. And that is the bivariate correlation; in the unique contribution the loss bites harder, because measurement error in the outcome swells the residual and widens the standard error, while error in the predictor attenuates the coefficient itself.

The check is dull and very persuasive: tabulate instrument, version, item count, scoring and α for your study and for the source study, side by side. If they differ, you are no longer speculating in the Discussion, you are reporting a fact anyone can verify by opening both papers. And if they match, you have ruled out an entire explanation and the remaining three carry more weight.

Explanation 3: it is not the same population

Range restriction is the silent version of this problem. If the source study sampled the general adolescent population and yours is a university cohort where nearly everyone is a heavy user, the standard deviation of the predictor narrows, and a correlation does not survive that: with a SD 30% smaller, a correlation of .30 in the broad population is observed as .22 in yours. The phenomenon has not changed; the slice of the range you are looking through has. Ceiling effects in the outcome and very different base rates do the same thing.

Then there is the part that can be a genuine difference. Age, setting, country, time of measurement, sample composition. If the original finding comes from 13-to-16-year-olds recruited through schools and you measured undergraduates, it is entirely possible that the effect exists in one group and not the other. That is not a failure of your study: it is a moderation hypothesis, and it is written as one, naming the specific characteristic you think moderates, rather than as a vague line about cultural differences that commits nobody to anything.

You check it with descriptives you already have. Compare the M and SD of the predictor in your sample against the source study's, usually sitting in their Table 1. Two lines of work, and the sentence it produces is one of the hardest to argue with that you can put in a Discussion: the standard deviation of the predictor in this sample was 30% smaller than in the source study.

Explanation 4: the original estimate was inflated

This one goes last on purpose, and not because it is unlikely. That source study, with N = 48 and a bivariate r = .34, t(46) = 2.45, p = .018, carries a structural problem: with that sample, the smallest correlation that can reach p < .05 is r = .28. No estimate below .28 ever leaves that design and reaches the literature. What you see published is not a random sample of real effects, it is the part of the distribution that cleared a filter, and the average of what clears a filter from above sits above the true value.

The size of that bias has been measured. In the Reproducibility Project (Open Science Collaboration, 2015), 100 psychology studies were replicated: of the 97 originals with a significant result, 35 were significant again, and the average effect size fell from r = .40 in the original studies to r = .20 in the replications. Roughly half. If the effect you were chasing was published once, with 48 participants and no preregistration, the reasonable expectation for your own study was never .34.

It goes last because it is the explanation you can least demonstrate from your own data and the one that reads worst when it comes first. Placed in the opening line of a Discussion it looks like you are pointing at other people to cover yourself. Placed at the end, after you have quantified power, measurement and population with your own numbers, it reads as a conclusion. The difference between the two versions is not the content, it is the order.

Three checks to run before you write a word

The sensitivity analysis. With your N, your number of predictors and your R², work out the smallest unique contribution your design could detect with 80% power. In the example it is sr = .21, and the effect you were chasing was worth .07. That single sentence answers half the questions a reviewer is going to ask, it is computed from the design rather than from the result, and it has nothing to do with post hoc power.

The two-column table. Report the zero-order correlation and the model coefficient side by side for every predictor. It is what lets a reader see at a glance that the bivariate effect is present and that what is missing is the unique part. Without that column, a reviewer reads non-significant and concludes your predictor is unrelated to the outcome, which is flatly untrue. If the source study reports a t or an F and you need its effect in the same metric as yours to compare them, convert it with the effect size from text calculator.

The equivalence test. If what you want to claim is that the effect is absent, a p value will not let you: absence of evidence is not evidence of absence. You need a TOST against a smallest effect size of interest that you set and justify (how to run an equivalence test). In the example, with an interval reaching .25, no reasonable equivalence test would let you conclude absence, and knowing that before you write saves you withdrawing the sentence once someone flags it.

How to write it up without sounding like an excuse

The rule fits in one line: an explanation without a number is an excuse. Possibly due to sample size is an excuse. A sensitivity analysis indicates that this design could only detect unique contributions of sr = .21 or larger with 80% power is a fact. Both sentences point at the same thing, but only one survives a response to reviewers.

Here is the full paragraph, with the four explanations in order and each anchored to something checkable: “Daily social media use correlated with depressive symptoms in the expected direction (r = .30, p < .001), but its unique contribution to the model was not significant (β = .08, 95% CI [-.09, .25], t(123) = 0.93, p = .354). The interval includes the coefficient reported by [Authors] ([year]), so these data do not contradict that finding but estimate it less precisely: a sensitivity analysis places the minimum detectable unique effect at sr = .21. In addition, social media use was measured here with a four-item short form (α = .66) rather than the 20-item scale used in the source study (α = .92), and the standard deviation of the predictor was 30% smaller, both of which reduce the observable magnitude. Replicating this test with 80% power would require approximately 1,040 participants.”

None of that belongs in Results. Results carries the coefficient, its interval and its p, with no adjectives and no promises. And no marginally significant, no approached significance: p = .354 is not a trend, it is a number that has already been reported and does not need dressing up (what to do when nothing comes out significant). The interpretation lives entirely in the Discussion, which is exactly where a reviewer goes looking for it.

And do not drop the predictor from the model. The hypothesis is written into your introduction; if it disappears from the results, the reviewer will notice, and what they will see is a model chosen after looking at the data. If you preregistered it, say so and leave it in place: a non-significant coefficient declared in advance adds credibility with an editor, while a spotless model nobody can trace the origin of removes it wholesale.

Frequently asked questions

What do I do when a predictor prior research says is significant does not come out in my model?

Before interpreting anything, check whether the published effect falls inside your confidence interval. If it does, your data do not contradict the literature, they just estimate it with less precision, and that is what you write. If it falls outside, you have a real discrepancy and you order the explanations: power for the unique effect, measurement differences, population differences, and last of all inflation of the original estimate by publication bias.

Can I drop a non-significant predictor from my regression?

Not if you announced it as a hypothesis in the introduction. The model you report has to be the one you planned, not the one that came out tidy. Removing a predictor after seeing its p value is choosing the model on the basis of the result, and a reviewer comparing your introduction with your results spots it in two minutes. A non-significant coefficient declared in advance costs you nothing; a model cleaned up afterwards costs you the paper's credibility.

How do I know whether my result contradicts the original study or is just underpowered?

Use the confidence interval of your coefficient, not the p value. If the source study's effect falls inside your interval there is no contradiction: your data are compatible with that effect and with zero. If it falls outside, the discrepancy is real. It also helps to report your design's minimum detectable effect at 80% power: if that is larger than the effect you were chasing, your study was never in a position to see it.

Is it acceptable to blame sample size for a non-significant predictor?

It is acceptable with a number attached and an excuse without one. Possibly due to the sample size says nothing. With N = 126 and f² = 0.0075, power for this test was 16%, and reaching 80% would require around 1,040 participants says exactly what happened. What does not work is computing observed power from your own effect: post hoc power is a transformation of the p value and carries no new information.

Can I say the original effect was inflated by publication bias?

You can, but it goes last and it goes with evidence. Useful evidence includes a small source sample (with N = 48, no correlation below r = .28 could reach significance), the absence of preregistration, or funnel plot asymmetry in the field's meta-analysis. Leading with it, before you have quantified power, measurement and population in your own data, reads as a defensive accusation.

How do I report a non-significant predictor in APA 7?

The same way as a significant one: coefficient, standard error or confidence interval, and the exact p value. For example, β = .08, SE = .086, t(123) = 0.93, p = .354. No rounding to p > .05, no marginally significant, and no adjectives inside the results section. The interpretation belongs in the discussion, not attached to the sentence where you give the number.

Keep the first check, because it changes the tone of everything you write afterwards: see whether the published effect fits inside your interval before you decide your study contradicts it. The four explanations follow from there, always in the same order and always with a number of your own attached to each. If what you are facing is a model with several correlated predictors and you are no longer sure which test actually answered your hypothesis, that is what my statistical consulting is for.