American Statistical Association (ASA) Issues “P-Value” Warning

There are too many ways to game clinical and scientific studies. Probably the most egregious is “Impact Factor” but close behind it is overreliance on the “p-value.” Several major journals have begun the process of diluting the importance of p-values and introducing other descriptive statistic measures to describe study validity and power.

Stepping authoritatively into the growing concerns over p-value is the American Statistical Association (ASA), which issued a statement on March 7, 2016, on the use of p-values in science. The statement said that researchers should be wary of over reliance on p-value.

According to the ASA, there already exists an overreliance on the p-value in scientific reasoning, and many investigators believe, often correctly, that obtaining a significant result of p < .05 is a “golden ticket” to publication. Furthermore, editors of scientific journals—and the major orthopedic journals are ALL guilty of this—too frequently reward studies with significant p-values without considering the validity of other aspects of those studies.

Why should we care? Because, say the scientists at the ASA, this overreliance may be partly to blame for the lack of replicability in science—especially where the measurement tools are subjective (based on the subjects’ own assessments) as they are in orthopedics or the psychological sciences.

“We hoped that a statement from the world’s largest professional association of statisticians would open a fresh discussion and draw renewed and vigorous attention to changing the practice of science with regards to the use of statistical inference, ” said ASA Executive Director Ron Wasserstein.

The ASA statement was the FIRST position on statistical practice EVER taken by the association and sets forth six principles that producers and consumers of scientific research should consider when evaluating p values:

1. A p-value can indicate how incompatible data are with a specified statistical model.
2. A p-value does not measure the probability that the studied hypothesis is true or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

According to the ASA and many leading scientific journal editors (so far none in orthopedics), manuscripts should not be accepted or rejected by a journal based on the p-value reported. Instead, urges the ASA, scientists, funders, journalists, and editors should evaluate the persuasiveness of the statistical argument as a whole.

Orthopedics This Week is increasingly concerned that overreliance on Impact Factor and a single statistical measure, p-value, is reducing the quality of orthopedic research appearing in the major orthopedics journals. To read our continuing discussions on the state of orthopedic journal research, please refer to the following articles which have appeared in Orthopedics This Week over the years:

1. Dump the ‘P’ Value December 15, 2015
2. Peer-Review Research Fails Reproducibility Test September 11, 2015
3. Contrition and the “BMP Issue”: NASS’s Bono Speaks October 6, 2015
4. Yale’s Study Challenges Carragee’s Mis-Measure of BMP-2 June 17, 2013
5. Carragee’s Mismeasure of rhBMP2 and Spine Surgeons October 17, 2011
6. Under Carragee The Spine Journal Lives Dangerously September 13, 2011