A New Look at P Values for Randomized Clinical Trials

NEJM Evidence, vol. 3, pp. EVIDoa2300003

Abstract

BACKGROUND: We have examined the primary efficacy results of 23,551 randomized clinical trials from the Cochrane Database of Systematic Reviews. METHODS: We estimate that the great majority of trials have much lower statistical power for actual effects than the 80 or 90% for the stated effect sizes. Consequently, “statistically significant” estimates tend to seriously overestimate actual treatment effects, “nonsignificant” results often correspond to important effects, and efforts to replicate often fail to achieve “significance” and may even appear to contradict initial results. To address these issues, we reinterpret the P value in terms of a reference population of studies that are, or could have been, in the Cochrane Database. RESULTS: This leads to an empirical guide for the interpretation of an observed P value from a “typical” clinical trial in terms of the degree of overestimation of the reported effect, the probability of the effect’s sign being wrong, and the predictive power of the trial. CONCLUSIONS: Such an interpretation provides additional insight about the effect under study and can guard medical researchers against naive interpretations of the P value and overoptimistic effect sizes. Because many research fields suffer from low power, our results are also relevant outside the medical domain. (Funded by the U.S. Office of Naval Research.)

Authors 6

1. Erik W. van Zwet Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

2. Andrew Gelman

   Columbia University

   Affiliation as printed

   Department of Political Science, Columbia University, New York

   Department of Statistics, Columbia University, New York

3. Sander Greenland

   University of California, Los Angeles

   Affiliation as printed

   Department of Epidemiology, University of California, Los Angeles, Los Angeles

   Department of Statistics, University of California, Los Angeles, Los Angeles

4. Guido W. Imbens

   Stanford University

   Affiliation as printed

   Graduate School of Business, Department of Economics, Stanford University, Stanford, CA

5. Simon Schwab
   Affiliation as printed

   Swisstransplant, Bern, Switzerland

6. Steven N. Goodman

   Stanford University

   Affiliation as printed

   Department of Epidemiology and Population Health, Stanford University, Stanford, CA