Of Mice and Men in Orthopedics

“So if you participate in our study, you may or may not undergo surgery. Are you interested?”

The patient—living in pain, mind you—would likely wonder if indeed this surgeon had all of his marbles.

Dr. Charles Fisher, a spine surgeon at the University of British Columbia, states, “In a perfect universe, orthopedic researchers would be able to assess the effectiveness of a given procedure by comparing those undergoing surgery and those not taken to the OR. We are dealing with humans beings, however, who quite naturally are reluctant to play roulette with their health.”

The most highly regarded form of research is the meta analysis, an “animal” rarely seen in orthopedics. Dr. Fisher, who also holds a Masters Degree in Health Care Epidemiology, explains, “In performing a meta analysis you take a number of studies with similar methodology and data, and combine them to obtain more study power. This could result in a very high quality effort if in fact you have a number of Level I, randomized controlled trials (RCT) on the same subject. In most surgical disciplines, however, there isn’t that degree of quantitative data that would make this possible.”

Christopher Bono, Chief of the Orthopedic Spine Service at Brigham and Women’s Hospital, concurs, and discusses the orthopedic researcher’s “Plan B.”

Because it is extremely difficult to pursue a true meta analysis in orthopedics—the exception being drug studies—the growing trend is to perform systematic reviews of the literature.

Such reviews, says Dr. Fisher, are increasingly popular because they are objective and are evidence based. “In the past, key thought leaders would be asked to write a review article and they would go perform a literature search and write the article. When it’s just one person doing this, though, bias naturally comes into play, starting with the fact that they are reviewing literature they want to include. Times have changed, however, such that there is now a transparent methodology with at least two reviewers, and standardized inclusion and exclusion criteria for the papers that will be reviewed. It will not give you the same power/numbers of a meta analysis, but it is much more objective and powerful than our research of the past.”

Breaking down the systematic review process, Dr. Bono says, “You (and at least one other researcher) first define the study question, i.e., ‘Does the operative treatment of hip fracture result in lower mortality?’ Then you do a literature review, preferably with the restriction that you will only include RCTs. While this usually limits you to a small number of articles, these are the highest quality studies. I recommend engaging the help of a medical librarian so that you can be confident that you have accessed all of the relevant databases and used all of the appropriate search terms. To the latter point, for example, if I am interested in lumbar fusion and perform a search only on the term ‘fusion, ’ then I would miss the articles that would be revealed by another search term, such as ‘arthrodesis.’”

Dr. Marcus GosslerHe continues, “Then you review the articles in detail, along the way distilling the information from the articles into a tabular format and assigning levels of evidence.”

Note that while each article already has a level of evidence, as part of this process you can downgrade them based on your review. For example, Journal X may say that a certain article is a Level 1 study, but because there was less than 80% follow up you determine that it is a Level 2.

“Afterwards, the team critiques the methodology, then looks at all of the articles and tries to answer the initial study question. Finally, you draw conclusions based on conflicting and/or supporting evidence.”

As for Dr. Fisher, he puts a fine point on the fine art of study selection. “When you are determining your inclusion and exclusion criteria, you should think about whether you want to include only studies with similar outcomes.”

For example, if you want to combine the data of two studies looking at the outcome of operations to reduce pain then you should ensure that both studies used the same outcome tool, i.e., the Visual Analog Pain Scale.

“Additionally, the patient populations should not be too dissimilar…it is not a good idea, for example, to combine 25 year olds with 80 year olds.”

And where can researchers go astray? Dr. Bono notes, “In defining the study question investigators may propose an unanswerable question, something that leads to misleading conclusions that can, for example, result in an insurer denying coverage for a procedure. For example, if I did a systematic review of the operative treatment of clavicle fracture nonunions, I would likely find no comparisons of operative and nonoperative treatment. The conclusion of this review would likely be that there is insufficient evidence to support operative treatment for clavicle nonunions. However, if you go back to the original question, namely, ‘Does operative treatment of clavicle nonunions result in better outcomes than nonoperative treatment?, ’ then you see that by definition it is already unsuccessful because it is a nonunion. This type of scenario highlights the importance of asking the right, and clinically meaningful, question.”

Sounds like something your average clinician wouldn’t be aware of? So true, says Dr. Bono. “While some clinicians are knowledgeable in this area, most need to partner with biostatisticians and/epidemiologists—people who are well versed in the methodology of systematic reviews. The clinician’s role is to keep everyone on track with regard to the definition and adherence to the study question. If the study question is defined by non clinicians then it can be quite off base.”

For example, I was recently on a conference call with representatives from multiple centers who want to do a systematic review about low back pain. Prior to obtaining any clinician input they said, ‘Let’s compare outcomes for disc herniation for a variety of treatments.’ They wanted to look at intradiscal electrothermal therapy and facet rhizotomy, both of which are not treatments for disc herniation.

“If they had proceeded with that route, then they would have of course found a lack of evidence—and they would have spent resources on something that was not clinically meaningful.”

Dr. Fisher sheds light on an organization that has spearheaded the move towards systematic reviews. “The UK-based Cochrane Collaboration, leaders in evidence-based medicine, are the ones who came up with the standard format for systematic reviews whereby you go through the literature and develop specialized research questions. By the end of 2009, there were 4, 027 published Cochrane Reviews on the Cochrane Database of Systematic Reviews in The Cochrane Library; and their method is very thorough. For example, the literature search is so exhaustive that it is not uncommon to begin with 3, 000 articles and end up with 60.”

In the end, it is important to remember the beginning. Dr. Fisher: “Everything always comes down to the research question. If it is too precise/narrow then you can’t easily generalize it to the entire population, meaning that it is not especially meaningful from a clinical standpoint. Obtaining external validity is really difficult in the surgical disciplines. You are comparing two surgical techniques, which, on the face of it may seem uncomplicated.”

Dig deeper, however, and you see other things that must be taken into account, namely, ‘What are the experience levels of surgeons? Fifteen years or two years?’ and ‘What is the pathway of care at one institution versus another (a lot of one-on-one attention, excellent rehabilitation facilities, etc.).’

He adds, “Many people think that a systematic review is the final verdict on a study issue/question, but that is not true if the review was not performed properly. It is similar to how when people see that something was a randomized controlled trial they tend to think, ‘Oh, this was an RCT…the results must be valid.’”

“When looking at systematic reviews, ” advises Dr. Bono, “remember that they are just like other studies…there are good and bad ones out there. Examining the methodology that was used before you take away any conclusions will help you avoid heading off in the wrong direction.”