In 2014, Uri Simonsohn, Leif Nelson, and Joseph Simmons published a paper that changed the way the scientific community thinks about the validity of published findings. Their proposal, p-curve analysis, starts from a seemingly simple but devastatingly incisive question: if the effects reported in the literature are real, the distribution of significant p-values should have a predictable shape. If it does not, something is wrong.
The logic of the p-curve
The p-curve is the distribution of statistically significant p-values (p < .05) reported across a set of studies. To understand its logic, one must recall a fundamental principle of statistics (and understand the problems with p-values): when a true effect exists, the probability of obtaining a very small p-value (for example, p < .01) is greater than the probability of obtaining a p-value just below the significance threshold (for example, .04 < p < .05). This is because the stronger the real effect, the more likely the statistical test is to produce compelling evidence against the null hypothesis.
From this principle, three fundamental scenarios follow. If the effects under study are real, the p-curve should be right-skewed: the tallest bar should be in the lowest p-value interval (.00-.01), and the bars should progressively decrease toward .05. If the effects are not real and the significant p-values are the product of chance, the p-curve should be flat, because under the null hypothesis all p-values are equally likely. And if researchers have engaged in p-hacking (manipulating analyses until p < .05 is achieved), the p-curve should be left-skewed, with an accumulation of p-values just below .05.
Difference from traditional meta-analysis
It is important to understand that p-curve analysis and traditional meta-analysis answer different questions. Meta-analysis estimates the combined effect size and its confidence interval, that is, it attempts to quantify how large the effect is. The p-curve, by contrast, does not estimate effect size: it evaluates whether the studies reporting significant results contain evidential value, that is, whether the distribution of significant p-values is consistent with the existence of a real effect or with the absence of an effect (or with p-hacking).
This distinction has crucial practical implications. A meta-analysis can produce a significant and reasonably large combined effect, but if the p-curve of the included studies is flat or left-skewed, there are serious reasons to doubt that the effect is genuine. The p-curve functions as a detector of p-hacking and selective reporting at the level of the study set, something that traditional meta-analysis is not designed to detect.
The three p-curve tests
The formal p-curve analysis involves three complementary statistical tests. The first is the right-skew test, which evaluates whether the p-curve contains more low p-values than would be expected by chance. If this test is significant, we conclude that the studies contain evidential value: something real lies behind the results. The second test is the flatness test, which evaluates whether the p-curve is consistent with the complete absence of an effect. If this test is significant, we conclude that the studies lack evidential value, which is consistent with a literature built on false positives.
The third dimension of the analysis uses a pp-value approach (p-values of p-values). For each significant study, the probability of obtaining a p-value as extreme or more extreme is calculated (given a true effect estimated from the data). These pp-values are combined using the Stouffer method to produce an overall test. This approach has the advantage of being more robust than simple visual inspection of the distribution bins, and it allows the strength of evidence to be quantified on a continuous scale.
Requirements and assumptions
The p-curve has specific requirements that limit its applicability. First, it uses only statistically significant p-values (p < .05), which means it discards all information contained in studies with non-significant results. This implies that a reasonable number of significant studies is needed for the analysis to have adequate power: with fewer than five or six p-values, results are uninformative. Second, each study must contribute a single p-value, which should correspond to the test of the main hypothesis, not a secondary or exploratory test. Selecting the correct p-value from each study requires judgment on the part of the researcher and must be documented transparently.
Additionally, the p-curve assumes that the included studies are independent and that each tests the same general hypothesis (though not necessarily an identical one). If studies test substantially different hypotheses, the interpretation of the p-curve becomes complicated because the expected distribution of p-values depends on statistical power, which in turn depends on the true effect size. If the true effects vary widely across studies, the shape of the p-curve can be difficult to predict even when all effects are real.
P-hacking and questionable research practices
The context that makes the p-curve especially relevant is the replicability crisis in psychology and other social sciences. P-hacking, the colloquial term for a set of questionable research practices that include selective data exclusion, exploration of multiple dependent variables, post-hoc addition of covariates, and deciding to stop collecting data when significance is reached, produces a particular distribution of p-values: an accumulation just below .05, that is, a left-skewed p-curve. When a body of literature shows this signature, the interpretation is that the significant results do not reflect real effects but rather the exploitation of researcher degrees of freedom.
The p-curve does not identify which individual studies are the product of p-hacking. It works at the set level: it evaluates whether the collection of significant results, taken together, is more consistent with the existence of real effects or with their absence. It is an epidemiological tool for science, not a forensic detector of individual fraud.
Practical use
To conduct a p-curve analysis, the most accessible approach is to use the web application p-curve.com, developed by the authors themselves. The application requires a list of statistical results in a standardized format (for example, t(45) = 2.31 or F(1, 78) = 5.42) and automatically produces the distribution, the formal tests, and an interpretable graph. For more customized analyses, the dmetar package in R includes functions for computing p-curves, and the pcurve package provides additional implementations. In any case, p-curve analysis should be considered a complement, not a substitute, for traditional meta-analysis, funnel plots, and formal tests of publication bias such as Egger's test. Together, these methods offer a multifaceted assessment of the credibility of a body of evidence that is far superior to any one of them individually.