I Don't Trust These Questionnaire Results: Five Checks Before You Analyze Them

The matrix is on screen and you do not believe it. Alpha comes back at α = .94 for a scale that struggled to clear .70 in the pilot. Two variables with no reason to move together correlate at r = .61. And scrolling down the file there is a row of threes from top to bottom, then a few more underneath. That feeling that something is off is usually right, and another test will not fix it. What fixes it is looking at five specific things in the raw data, all of them before the first analysis. Here they are in order, with the threshold each one uses, what it does to your results if you let it through, and the sentence you write in the manuscript afterwards.

The item-total correlation: the reverse item nobody recoded

The corrected item-total correlation is the correlation between one item and the sum of the other items on its scale, with that item left out of the sum. The correction is what makes it useful: leave the item in and it correlates partly with itself, so everything looks stronger than it is. SPSS gives it under Analyze > Scale > Reliability Analysis with the Scale if item deleted statistics box ticked; in R it is the r.drop column from psych::alpha(); jamovi calls it the item-rest correlation in its reliability module. What you are looking for first is not a healthy number, it is a sign. On a scale whose items all measure the same thing, a negative correlation means that item runs the opposite way to the rest.

Almost always it runs the opposite way because it is reverse-worded and nobody recoded it. The recode is (theoretical maximum + theoretical minimum) − score: on a 1 to 5 Likert that is 6 − x, and on a 0 to 4 scale it is 4 − x. Using maximum − x on a scale that starts at 1, which is the common slip, leaves a zero where a one should be and shifts the whole scale down a point. Once that is right the check takes seconds: recode, ask for the table again, and no item-total correlation is allowed to stay negative. The full pre-analysis step, metric and within-scale missing values included, is in reverse items, sum or mean, and missing items.

It is worth seeing how much damage this does, because it explains the strange alpha that brought you here. Take a 20-item scale, 12 positively worded and 8 reverse-worded, with a mean correlation of .35 inside each block. Leave those 8 unrecoded and they correlate at −.35 with the other twelve, the mean inter-item correlation for the whole scale collapses to zero, and alpha comes out at α = −.08. A negative alpha is not a software glitch or an odd sample: with items pointing the same way it is arithmetically impossible, so it always means part of your scale points the other way. In the milder version, three unrecoded reverse items out of ten leave alpha near α = .17 when the properly recoded scale sits around α = .81, with every single datum present.

With everything recoded, the usual convention asks for a corrected item-total correlation of .30 or above (Nunnally and Bernstein, 1994); between .20 and .30 the item is weak, and below .20 it shares almost nothing with its neighbors. That is a convention rather than a law, and it has an obvious limit: deleting items until alpha climbs is a fast route to a shorter, more redundant scale stripped of the content that justified using it. If the item is theoretically central, you report that it behaved badly in this sample rather than quietly dropping it. How the full table is written up is in how to report an item analysis in APA 7, and if all you want is the table of item-total correlations with alpha-if-deleted, the item analysis calculator produces it.

Straight lines: the participant who ticked the same box all the way down

Open the raw file and read rows, not columns. A row with the same number repeated forty times is not an unusually consistent person: in a survey that mixes three scales, different anchors and a few reverse items, it is somebody who stopped reading at item five. The literature calls it insufficient effort responding, and prevalence estimates sit around 10% to 12% in undergraduate samples with wide variation between studies (Meade and Craig, 2012). With online panels and course-credit participant pools in the mix, assuming your sample contains none of them is optimistic.

Four signals are enough to catch them, and all four come out of data you already have. Long string. The longstring index (Johnson, 2005) is the longest run of identical consecutive responses for a person, and it has to be read against the structure of the questionnaire, because twenty consecutive items from the same subscale can legitimately get the same answer. Within-person variability. The standard deviation of each participant's answers across the questionnaire: zero is the perfect flat line, and very low values are the flat line with a wobble. Time. Huang et al. (2012) propose a cut-off of two seconds per item; below that there is no physical time to read and answer. Psychometric synonyms and antonyms. Meade and Craig (2012) pair near-identical or near-opposite items and correlate them within each person: a genuine respondent shows a high positive correlation across the synonym pairs.

What those rows do to your numbers depends on where they parked. If some tick 1 throughout and others tick 5 throughout, each of them contributes the same score to every item and pushes the covariance between items up: your alpha improves and so do your correlations, with nobody having measured anything. If they all sit on the midpoint they contribute zero variance and the correlations are attenuated instead. And where there are reverse items, a flat line breaks them in both directions at once. In all three cases the number in front of you is no longer about your construct, and the problem is not that it is high or low, it is that it does not depend on what you asked.

The rule of use is what separates data cleaning from dredging. Set the criterion before you look at the hypotheses: which signals count, how many flags justify exclusion, and what happens to borderline cases. Then report how many were excluded, why, and, for a demanding reviewer, the result with and without them. Dropping participants after seeing that the effect did not come through, then rerunning the p value, is a different activity and it shows from the outside: the final sample does not match the recruited one and nothing explains the gap. An explicit attention check ('please select Agree for this item') helps, but it does not replace the indices, because some people pass it by chance and some fail it through a lapse while answering everything else properly.

A high alpha does not prove the scale is good. It proves it is long

Cronbach's alpha rises with two things: how much the items share, and how many items there are. The second one is the trap. With a mean inter-item correlation of .15, which is the floor of what is recommended, 40 items give α = .88. With a mean inter-item correlation of .50, which is the ceiling, 6 items give α = .86. The weak scale beats the good one by two hundredths purely by being seven times longer, and if alpha is all you report nobody can tell them apart. So that α = .94 on a 45-item measure does not say what it looks like it says: it says the scale is long, and you still do not know whether it measures one thing or four.

The number that does discriminate is the mean inter-item correlation, and it fits on the same line of the manuscript. Clark and Watson (1995) recommend it sit between .15 and .50, and toward .40 to .50 for a narrow construct: below that the scale is measuring scattered things, above it the items are the same item rewritten. The classic demonstration is Cortina (1993), who showed that an 18-item scale made of three dimensions that do not even correlate with each other can clear α = .70 comfortably. A high alpha is entirely compatible with a multidimensional scale; all it guarantees is that there is enough covariance among the items, not where that covariance comes from.

Alpha also carries an assumption almost nobody checks: tau-equivalence, meaning every item loads equally on the construct. When that fails, and it usually does, alpha underestimates reliability, and the alternative is McDonald's omega, estimated from the factor model without that constraint. On a decent unidimensional scale the two values look alike; when they diverge sharply, omega is the one telling you something. Everything that has to be declared alongside the coefficient (which statistic, which sample it was computed in, the interval if you give one) is in how to report Cronbach's alpha in APA 7.

There is a ceiling on this too. An α = .97 on a twenty-item scale is not excellent news, it is redundancy: the items are asking one question in different words, and the scale is spending four minutes of your participant's attention collecting what six items would collect. The practical version: report alpha, report the number of items and the mean inter-item correlation next to it, and if the construct is multidimensional report the coefficient per subscale rather than one global value that averages across dimensions and hides them.

If you measured everything the same way, part of that correlation is method

The correlation that bothers you has an obvious suspect when both variables come from the same place. A fifteen-minute online form, one person reporting on perceived stress, supervisor support and exhaustion, the same five-point anchors across all three scales, all in one sitting. Part of what correlates is the constructs and part of it is having asked everything together: the pull to look consistent with what you just answered, social desirability, acquiescence, and the negative affect that colors the report of the stressor and the report of the symptom at once. It is called common method variance and it has been cataloged since Podsakoff et al. (2003).

The reflex answer is Harman's single-factor test, and that reflex no longer works as a defense. It has no null hypothesis and no critical value, the 50% threshold rests on nothing, and when it has been simulated it only detects bias at magnitudes that do not occur in real studies. Passing it proves nothing, and the reviewer asking you to address method bias usually knows that. What sits behind it, and what goes in its place, is in common method bias and Harman's test.

For today's check, one question and two consequences will do. The question: is there a single variable in my study that does not come from the same questionnaire, the same person and the same moment? If the answer is no, every correlation in your table shares a source of covariance that is not theoretical. The first consequence is that correlations between self-reports collected together are not comparable with those from studies that separated sources or occasions, and cannot be discussed as if they were. The second is that the finding least likely to survive this is the smallest one: if your conclusion rests on r = .18 between two scales from the same form, it rests on the most fragile part of the table.

What genuinely reduces the bias is design, and it has to be decided in advance: separate predictor and criterion in time, change the source of at least one variable (a record, another informant, a behavioral task), protect anonymity properly to lower evaluation apprehension, and stop every scale from sharing the same format and anchors. If the data are already collected, the honest answer is not a passed test: it is naming the limitation, saying which design remedies you did apply, and not selling as robust a result that rests on a single source.

Floor and ceiling: when the scale has nowhere to put your sample

A floor or ceiling effect is what you have when a large share of the sample piles up on the lowest or highest possible score. The most cited criterion puts it at 15%: if more than that obtains the minimum or the maximum, the instrument is taken to have the problem in that sample (Terwee et al., 2007). And it is in that sample, not in the instrument forever: a depression measure built for clinical populations behaves perfectly in a clinic and stacks on the floor once you hand it to second-year undergraduates, where a large share answer the minimum on nearly every item.

What it does to your results is quiet and systematic. It restricts range, and with restricted range correlations attenuate: a real relationship shows up smaller than it is, and sometimes disappears. It skews the distribution hard, which you then try to fix with a transformation, treating the symptom. In repeated measures it removes room to move: someone already at the maximum cannot improve, so your intervention looks weaker than it is, and someone at the minimum cannot deteriorate, so a control group looks artificially stable. At item level, an item where 90% choose the same option has a tiny standard deviation and drags its own item-total correlation toward zero even when the item is perfectly well written.

Seeing it takes two minutes and almost nobody does it, because the mean and the standard deviation hide it. Ask for the frequency table of every item and look at where people bunch up; compute the percentage of participants at the lowest possible and the highest possible total score; and scan the item-level standard deviations for the ones that collapse. A histogram of the total score helps, but the fine diagnosis is at item level: a scale whose total looks reasonable can contain four items at ceiling.

When it turns up, you report it and discuss it rather than hiding it. The useful sentence gives the exact percentage at the floor, acknowledges that correlations with that variable are attenuated, and explains that the instrument was not built for the range this population occupies. If you are still at the design stage and can choose, the fix is an instrument with more room at the end where your sample lives, not adding response options to a validated scale, which changes the metric and leaves you unable to compare with anyone.

The order of the five checks, and what you report

Run them in this order, because each one clears the ground for the next. One. Rows: response times, long strings and within-person variability, with the exclusion criterion fixed before you look at anything else. Two. Reverse items: recode, then confirm that no item-total correlation stays negative. Three. Item distributions: frequencies, standard deviations, and the percentage at the minimum and the maximum. Four. Reliability, now on clean data and in the right order: alpha or omega, with the number of items and the mean inter-item correlation beside it. Five. Method: where each variable comes from and how much they share simply by coming from the same place. Computing alpha before recoding is the classic mistake behind 'this scale does not work in my sample'.

The manuscript does not need a new section for any of it: six sentences across Method and Results will do. The response-quality exclusion criterion and how many cases went. The range of corrected item-total correlations. The reliability coefficient with the number of items and the mean inter-item correlation. The percentage at floor or ceiling when it passes 15%. And an honest limitation if everything came from one self-report collected in one sitting. None of that costs space, and all of it is what a reviewer uses to decide whether to trust the rest of the paper.

A reviewer who cannot find those sentences does not conclude that the checks were done badly. They conclude that they cannot tell whether the checks were done at all, and they read your results differently from that point on. That is a paragraph of difference, not a study of difference. Reporting the mean inter-item correlation or the percentage at the floor is not a claim that your data are perfect: it shows you looked at them before believing what came out.

Frequently asked questions

How do I know if an item was never reverse-scored?

The corrected item-total correlation tells you. On a scale whose items measure the same thing that correlation has to be positive; a negative value (r = −.38, say) means the item runs the opposite way to the rest, and it is nearly always a reverse-worded item that nobody recoded. The second clue is alpha itself: an absurdly low or outright negative value on a published scale with a good track record points to the same thing. Recode with (theoretical maximum + theoretical minimum) − score, then ask for the table again to confirm.

What is an acceptable item-total correlation?

The common convention asks for .30 or above on the corrected item-total correlation (Nunnally and Bernstein, 1994). Between .20 and .30 the item is weak and needs to be read against the theory; below .20 it shares almost nothing with the other items. It is a convention rather than a demonstrated threshold, and it does not license deleting items until alpha climbs: if the item is central to the construct, you report that it behaved badly in this sample and discuss it instead of dropping it silently.

Can I delete participants who gave the same answer to everything?

Yes, if the criterion was fixed before you looked at the hypotheses and you report it. The standard approach combines signals: a long run of identical responses, a within-person standard deviation near zero, a response time under two seconds per item (Huang et al., 2012), and failed attention checks. Two or more flags, with the criterion written down in advance, make the exclusion defensible. What is not defensible is cleaning the sample after seeing that the effect did not appear; if you do it late, report the result with and without the excluded cases.

Is a Cronbach's alpha of .95 good?

It is a redundancy warning. A very high alpha on a long scale usually means the items ask one question in different words, or simply that there are many of them: 40 items with a mean inter-item correlation of .15 already give α = .88 without the scale measuring anything well. The usual reference band runs from .70 to .90, and above .95 the item content is worth rereading. Always report alpha with the number of items and the mean inter-item correlation beside it, because that is what makes it interpretable.

When does a questionnaire have a ceiling effect?

The most cited criterion is 15%: more than 15% of the sample obtaining the highest possible score means a ceiling effect, and the same at the lowest score means a floor effect (Terwee et al., 2007). It is a property of the scale in your sample, not of the instrument in the abstract. When it appears, correlations with that variable are attenuated through range restriction, the distribution turns sharply skewed, and in pre-post designs the room to change disappears.

How do I know whether my correlations are common method variance?

No test separates it out, and that is the point. If every variable comes from the same questionnaire, the same person and the same moment, part of the covariance is method, and no post hoc procedure quantifies how much reliably; Harman's single-factor test in particular is not evidence of absence. What you can do is acknowledge the limitation, offer a marker variable chosen a priori or a common latent factor as partial evidence, and avoid resting your headline conclusion on the smallest correlation in the table.

With the reverse items recoded, the straight lines removed under a written criterion, alpha reported next to its mean inter-item correlation and the floor declared, you know what your questionnaire measures in your sample and what it does not. If what survives the five checks is a scale that genuinely does not work here, rather than a recoding slip, that is no longer data cleaning but a measurement problem: it belongs in the Method section and in the limitations, not buried inside the analysis.