Common Method Bias: Harman's Test, Why It No Longer Works and What to Use Instead

You collect everything in one twenty-minute online questionnaire, the same five-point Likert anchors throughout, one person answering for all three variables. The manuscript goes out, and round one brings the familiar sentence: the study relies on single-source, single-wave self-report data, please address common method bias. For fifteen years the standard reply was a paragraph about Harman's test and on with the day. That reply is dead, and it pays to understand why before you write anything, because the reviewer asking usually knows the literature that killed it. Here is what the bias is, why Harman does not detect it, what the common latent factor and the marker variable really do, which design remedies work, and how to write the response when the data are already sitting on your hard drive. The wider catalog of traps is in common errors when interpreting statistical results.

What common method bias is, and what it is not

Common method bias is the share of covariation between your variables that comes from the measurement method rather than from the constructs you think you are measuring. The reference definition is Podsakoff, MacKenzie, Lee and Podsakoff (2003): variance attributable to the measurement method rather than to the constructs the measures represent. Applied to your design, if you ask about workload, headteacher support and emotional exhaustion in one questionnaire, from one person, on the same five-point anchors, inside the same ten minutes, part of what correlates is the constructs and part is the fact that you asked it all at once.

The sources are well cataloged and there is more than one. From the respondent come the consistency motif (people want to look coherent with what they have just answered), social desirability, acquiescence, and stable differences in negative affectivity that color the report of the stressor and the report of the symptom alike. From the items come the common response format, common anchors, ambiguous wording and double-barrelled phrasing, all of which nudge everyone toward the same end of the scale. From the item context come order and priming effects: what you ask first changes how the next thing is answered. And from the measurement context comes the obvious one, measuring everything at the same moment, in the same place, through the same medium.

It is worth being clear about what this does to your correlations, because the folk version is simpler than the reality. The bias can inflate them, which is the case that worries reviewers, but it can also attenuate them when the method pushes the two measures in different directions, or when reverse-worded items drag acquiescence the other way. The one thing it cannot do is manufacture an interaction effect out of nothing, and that turns out to be more useful than it sounds. We will come back to it.

What it is not: grounds for writing off any study that uses self-report. Spector (2006) called it an urban legend outright, and the argument still stands. Some constructs exist only in the first person — perceived support, intention to leave, satisfaction — and measuring them through an external informant does not reduce the bias, it changes the construct. The useful question is never whether method variance exists, because it always does, but how much, where, and whether it is enough to explain the effect you are defending.

Why Harman's single-factor test no longer works

The procedure is familiar. You put every item from every scale into one exploratory factor analysis, extract without rotation, and check two things: whether a single factor emerges, and whether the first one accounts for more than 50% of the variance. If neither happens, you write the sentence about no common method bias being detected and move on. There is a confirmatory variant that fits a one-factor model and shows it fits badly. Both share the same faults.

The first is that it is not a test. There is no null hypothesis, no reference distribution, no critical value and no statistical decision — there is a percentage and a round threshold that comes from no demonstration at all. The second is a matter of attribution, and it explains a good deal: Harman did not propose this. The procedure spread through Podsakoff and Organ (1986), who cited Harman's (1967) factor analysis textbook for the single-factor technique rather than for any bias diagnostic. We have spent forty years naming it after someone who never wrote it.

The third fault is the fatal one, and it is empirical. With twenty-four items spread across four correlated constructs, the first unrotated factor almost never clears 50%: in the burnout example running through this article, 298 secondary school teachers, it accounted for 29.8% of the total variance, and that figure would have looked much the same with severe bias or with none. Fuller et al. (2016) checked this by simulation and their conclusion is the one to cite — the procedure only signals bias when its magnitude exceeds anything a real study carries. A diagnostic that never comes back positive carries no information. Passing it is not evidence that bias is absent; it is evidence that you ran it.

The fourth is that even if it did detect something, it corrects nothing: there is no follow-up step, no adjusted estimate, no decision to take on the result. The confirmatory version is just as empty. A one-factor model fitting worse than your four-factor model is a practically guaranteed outcome once your scales have any discriminant validity at all, and it says nothing about whether an additional method factor is operating on items that also measure genuinely different constructs. Nobody ever claimed method variance was the only variance in the data.

The common latent factor: what it does and why it still does not settle it

The common latent factor, or unmeasured latent method construct, is a serious step up. You add to the measurement model a latent factor with no indicators of its own that loads on every item in the study, usually with those loadings constrained equal so the model is identified, and you compare the solution with and without it. What you inspect is the standardized loadings of each item on its own construct, the correlations among factors and, if you like, the fit. If none of that moves, the argument is that method is not doing the heavy lifting.

Numbers make it concrete. In that same example, the four-factor measurement model fitted acceptably, χ²(246, N = 298) = 402.18, p < .001, CFI = .944, RMSEA = .046, 90% CI [.039, .053], SRMR = .051. Adding the common latent factor gave a standardized common loading of .39, meaning method accounted for roughly 15% of item variance, and loadings on the substantive constructs dropped by between .02 and .08, none falling below .50. For reference, classic reviews put the median method variance in self-report measures at around a quarter (Williams, Cote, & Buckley, 1989), so 15% is not an alarming figure. The convention for reporting the measurement model itself first is covered in how to report a CFA in APA 7.

The problem is conceptual and no amount of extra sample fixes it. The method factor is unmeasured, so you do not know what it is: it soaks up method variance and equally any substantive general factor your constructs share for theoretical reasons. In a wellbeing study where stress, anxiety and exhaustion correlate because a general distress factor sits underneath them, the common latent factor eats that distress and hands you back deflated estimates that look clean. Richardson, Simmering and Sturman (2009) tested this by simulation and their verdict is uncomfortable: correcting with an unmeasured method construct can leave you worse off than not correcting at all. On top of that sit the usual practical troubles, non-convergence and negative variances as soon as the model grows.

If you use it anyway — and that is defensible — report the whole thing. Give the loadings with and without the factor, the magnitude of the method variance, and the construct correlations under both solutions, rather than the free-floating sentence about controlling for bias with a common latent factor, which lets nobody check anything. And decide before you look at the results whether the factor is retained in the structural model: if it is, every coefficient you publish comes from the model with the method factor in it, and your analysis section has to say so rather than leave it implied.

The marker variable, done properly

Lindell and Whitney (2001) took a different and more elegant line: instead of inventing an unmeasured factor, you measure something. You include in the questionnaire a marker variable with no theoretical relationship to any construct in your study, measured with the same method, the same response format and in the same session. Since its true correlation with everything else should be zero, whatever you observe above zero estimates the shared method variance, and you partial it out of the substantive correlations.

The arithmetic fits on one line. With an observed correlation between workload and emotional exhaustion of r_U = .41 and a marker correlation of r_M = .11, the adjusted correlation is r_A = (r_U − r_M) / (1 − r_M) = .34, which at N = 298 remains clearly different from zero. Reporting both the observed and the adjusted value, alongside which marker you used and why, is more informative than any Harman paragraph you have ever written.

The catch is in how the marker is chosen, and it is where almost everyone falls down. It has to be chosen a priori, decided before data collection and deliberately built into the battery. Simmering, Fuller, Richardson, Ocal and Atinc (2015) reviewed published practice and found that most markers were picked afterwards from whatever happened to be in the survey, often through the shortcut of taking the smallest positive correlation in the matrix. That shortcut is a downwardly biased estimator: the minimum correlation in a matrix is by construction the cell carrying the most negative sampling noise, so you understate the bias precisely when you are trying to show there is none.

The modern version is the CFA marker technique of Williams, Hartman and Cavazotte (2010), which brings the marker into the measurement model and compares a series of nested models instead of doing a subtraction. The baseline model fixes the marker's effect at zero; Method-C lets it act with equal loadings across all items; Method-U frees them; Method-R returns the construct correlations to their baseline values to see whether that worsens fit. In the teacher example, Method-C improved on the baseline, Δχ²(1) = 7.31, p = .007 — there is a method effect — Method-U added nothing, Δχ²(23) = 27.94, p = .218 — the effect is congeneric, equal across items — and Method-R did not worsen fit either, Δχ²(6) = 7.45, p = .281, meaning the relationships among constructs survive the control. That chain of nested comparisons is exactly what the SEM model comparator runs, and if you work in PLS the quick equivalent is Kock's (2015) full collinearity assessment, with a VIF of 3.3 as the warning line.

The remedies that work are design, not analysis

Temporal separation. You measure the predictor at T1 and the criterion two weeks or a month later. It cuts the consistency motif, priming and most of the mood of the moment, and it is the remedy a reviewer accepts without argument. You pay for it in attrition, and you should know what it does not fix: stable respondent tendencies — acquiescence, social desirability, negative affectivity — are still there at both waves, because they belong to the person rather than the sitting.

Different sources. The most powerful remedy and the most expensive. Breaking the chain at a single link is enough: patient self-report for symptoms and a structured clinical interview for diagnosis, pupil self-report for classroom climate and a class teacher's report for behavior, a perceived stress questionnaire and unit-level absence records for the outcome. You do not need to change the source of every variable; once the dependent variable comes from somewhere else, the common method explanation stops being plausible for your main effect.

What you can still do inside a single questionnaire. Proximal separation, putting predictor and criterion in different sections with pages in between; changing format and anchors between blocks so the response pattern is not inherited; counterbalancing scale order across participants; genuine anonymity guaranteed in writing, plus the explicit instruction that there are no right answers, which is the known lever against evaluation apprehension; and a pass over item wording to strip out ambiguity, jargon and double-barrelled phrasing. It is all in Podsakoff, MacKenzie and Podsakoff (2012), and none of it costs money.

The interaction argument. Siemsen, Roth and Oliveira (2010) demonstrated something almost nobody uses and that is worth having in a response letter: common method variance cannot generate a spurious interaction effect. If anything it attenuates one, so a significant moderation term in single-source data is an effect that survived the bias rather than one created by it. It applies to moderation and to quadratic terms, not to main effects or simple mediation; if your central hypothesis is a moderation, that paragraph is worth more than any post hoc test. How to build and report that model is in mediation and moderation with PROCESS.

How to answer the reviewer once the data are collected

What does not work is the Harman paragraph capped with a claim that common method bias is therefore not a problem in this study. Conway and Lance (2010) wrote the piece to keep at hand here, because it addresses exactly that situation: what a reasonable reviewer should expect is not a passed test but an argument for why self-report is the appropriate method for those constructs, evidence of construct validity for the scales, and a description of the design remedies you actually applied. Bias is not passed or failed, it is bounded.

A model paragraph you can adapt by changing the details: “We thank the reviewer for this point. The data are single-source and single-wave, so common method bias cannot be ruled out, and we do not present any post hoc procedure as though it ruled it out. We have added three things. First, the design included proximal separation — the predictor and criterion scales appeared in separate sections and used different response formats — guaranteed anonymity, and an explicit instruction that there were no right answers, following Podsakoff et al. (2012) on reducing evaluation apprehension. Second, we included a marker variable built into the battery a priori and theoretically unrelated to any construct in the model; the CFA marker technique of Williams et al. (2010) indicates a small, congeneric method effect, and the construct correlations hold once it is controlled (Table S2). Third, the central hypothesis is an interaction, and common method variance cannot generate spurious interaction effects, only attenuate them (Siemsen et al., 2010). All of this is now in the limitations, and we have removed Harman's single-factor test from the previous version, given its insufficient sensitivity (Fuller et al., 2016).”

In the manuscript itself, the limitation is written short and without alibis: “All measures were self-reported and collected in a single session, so part of the observed covariation may reflect common method variance. The marker procedures applied suggest a small method effect but do not rule it out; replication with temporally separated or multi-source measures remains necessary.” A reviewer reading that sees someone who understands their own design. A reviewer reading that no common method bias was detected sees someone who copied a paragraph.

Then there is consistency, which settles more rounds than you would think. Do not call it a test when it is not one. Do not say you controlled for the bias when you only estimated it. If the common latent factor stays in the final model, every estimate you report comes from that model and the analysis section has to say so. And if you picked the marker after seeing the data, say that, and label it exploratory — passing it off as a priori is the kind of detail a reviewer checks against the questionnaire you uploaded as supplementary material.

Frequently asked questions

What is common method bias?

It is the part of the correlation between two variables that comes from measuring them with the same method rather than from any real relationship between the constructs (Podsakoff et al., 2003). It shows up mainly when one person answers everything, at one moment, in one response format. Its sources run from the consistency motif and social desirability through to common anchors and item order effects. It can inflate correlations and it can also attenuate them, though inflation is what worries reviewers.

How do you run Harman's single-factor test in SPSS?

Under Analyze > Dimension Reduction > Factor, you enter every item from every scale, choose principal components or principal axis extraction, fix the number of factors at 1 or request the unrotated solution, and read the variance accounted for by the first factor. It takes two minutes, and that convenience is part of the problem: the procedure spread because it is cheap, not because it works. If you do run it, know that a result below 50% does not demonstrate that bias is absent.

Is Harman's single-factor test still acceptable?

Not as evidence that bias is absent. It is not a statistical test — no null hypothesis, no reference distribution, no critical value — the 50% threshold is a convention with nothing behind it, and the simulations in Fuller et al. (2016) show it only flags a problem when the bias is larger than in any real study. Journal editorials now discourage it explicitly, following the argument of Conway and Lance (2010). If you include it, do not present it as proof of anything; in most cases you are better off replacing it.

What are the alternatives to Harman's test?

Two post hoc and one by design. The common latent factor adds an unmeasured method factor loading on every item and compares the solution with and without it. The marker variable (Lindell & Whitney, 2001) measures something theoretically unrelated to your model and partials out its correlation; its modern form is the CFA marker technique of Williams et al. (2010), built on nested models. And the alternative that genuinely reduces the bias is design: temporal separation, different sources and real anonymity.

How do you choose a marker variable?

A priori, before data collection, with a theoretical justification for why it should not correlate with any of your constructs. It has to be measured with the same method, the same response format and in the same session, otherwise it does not capture the method you are trying to estimate. What to avoid is the shortcut of taking the smallest positive correlation in the matrix: it understates the bias by construction, because that cell is the one carrying the most negative sampling noise.

How much common method variance is too much?

There is no defensible cut-off, and be wary of anyone who offers one. As a descriptive benchmark, classic reviews put the median method variance in self-report measures at around a quarter (Williams, Cote, & Buckley, 1989), so a method factor accounting for 15% or 18% of item variance sits within the expected range. What decides the matter is not the percentage but whether your conclusions change once you control for it: if the construct correlations hold, you have an argument; if they collapse, you have a different finding from the one you thought you had.

With the limitation acknowledged, a marker chosen a priori or a latent factor reported in full, and the design remedies you did apply written down, that reviewer request stops costing you a round. If your problem is not the wording but how to fit the measurement model with the marker inside it, that is what my statistical consulting is for.