High Rate of Misreporting in Pediatric Orthopedic Literature

A new multicenter study led by Hospital for Special Surgery (HSS) has found that the pediatric literature has a depressingly high rate of misreporting.

The study, “How Often Are Study Design and Level of Evidence Misreported in the Pediatric Orthopaedic Literature?,” appears in the November 22, 2019 edition of the Journal of Pediatric Orthopaedics.

Drake LeBrun, M.D., M.P.H., a second year orthopedic surgery resident at Hospital for Special Surgery and co-author on the work explained the genesis of this study to OTW, “In our routine review of the literature, we were noticing high rates of study design misclassification across journals and subspecialties. Such ‘misreporting’ can lead to erroneous interpretations of a study’s findings, which can in turn lead to poor clinical judgments.”

“Once we realized the potential extent of the problem of study design misclassification, we came up with the current study as a way to more systematically assess the degree of study misclassification in a large sample of the orthopaedic literature.”

“First, the majority (52%) of studies simply did not report a clear study design. This makes it challenging for orthopaedic clinicians to understand how the study was conducted, what types of conclusions can be drawn from the data, and what types of bias might affect the results.”

“Secondly, nearly half (48%) of studies in our sample did not report a level of evidence [LoE]. Level of evidence is a helpful—albeit limited—metric of a study’s relative strength. Not reporting a level of evidence has a couple implications.”

“It makes it harder for orthopaedic surgeons to evaluate the strength of the data they are using to make clinical decisions. It also makes it harder for researchers conducting systematic reviews and meta-analyses to synthesize the data in the context of the overall literature.”

“In an era in which meta-analyses allow us to synthesize multiple data sets into broader samples and more rigorous data, it is imperative that the raw data be as clearly and accurately described as possible.”

“Thirdly, among the studies that did report a study design and/or level of evidence, 19% reported the wrong study design and 28% reported the wrong level of evidence. Interestingly, of the studies that misreported their level of evidence, only 35% overreported their level of evidence while 65% underreported (i.e., effectively ‘underselling’) their level of evidence.”

“A few steps could be taken to reduce the rate of study design and level of evidence non-reporting and misreporting. First, it would be helpful if all published observational studies explicitly reported study design and level of evidence according to standardized criteria (e.g., STROBE [Strengthening the Reporting of Observational studies in Epidemiology] criteria, JBJS [Journal of Bone and Joint Surgery] level of evidence guidelines, etc.). Similarly, training reviewers and editors to cross-check a paper’s stated study design and level of evidence would help to ensure that published studies are correctly classified.”

“The use of vague or contradictory terminology should be discouraged. Terms like ‘comparative study,’ ‘retrospective review,’ or ‘chart review’ are vague and not part of standardized epidemiological jargon. These terms could refer to any number of observational study designs and should be discouraged in favor of standardized study designs (e.g., case-control study, retrospective cohort study, etc.) that are consistent throughout the medical literature.”

“Lastly, I’d argue that the differences between observational study designs are relatively straightforward. Orthopaedic investigators should have little difficulty understanding these differences if they were appropriately trained to do so. To that end, it’d be helpful to include a critique of study design and LOE in standard journal clubs and other research-oriented reviews among orthopaedic residents.”

“We would like orthopaedic surgeons to critically evaluate the literature based on an accurate understanding of different observational study designs, and what each type of study design allows one to conclude. Overall, more rigor in evaluating observational studies is required on the part of orthopaedic investigators, reviewers, and journal editors to ensure that the data we use to make clinical decisions is valid and clear.”