Study: Track Orthopedic Trainees’ Errors!

Johns Hopkins researchers have found that it is necessary to track the errors of orthopedic trainees—not just assess what they are getting right. The research was led by Dawn LaPorte, M.D., a professor of orthopedic surgery at the Johns Hopkins University School of Medicine.

As indicated in the news release, “The team then tested the [Objective Structured Assessment of Technical Skills, or OSATS], in residents practicing three different approaches to access a shoulder in need of repair: from the front, back or side…”

“Three orthopaedic specialists from Johns Hopkins then used the grading systems to evaluate the work of 23 Johns Hopkins medical residents ranging from their first to fifth year of residency as they performed each of the procedures on cadavers. Under OSATS, the residents received a point for the successful completion of each step in the checklist and a zero for failed or incomplete steps. They also received zeros if they performed the steps out of order…Besides the OSATS checklists, the faculty surgeons rated residents using the so-called Global Rating Scale and a simple pass/fail system.”

“The Johns Hopkins team found that OSATS and the Global Rating Scale provided good, objective ways of measuring resident performance, while the pass/fail system gave residents unambiguous feedback. As expected, more advanced residents received higher OSATS and Global Rating Scale scores than those just beginning their residencies…Across all three procedures, examiners observed 11 incidents in which residents damaged the nerves or veins. First- and second-year residents were responsible for nine of those mistakes.”

“Crucially, however, the Johns Hopkins team found that none of the three evaluations adequately captured those mistakes. The pass/fail scheme came closest, in that residents received a failing score for severing a nerve or major blood vessel, but it is not set up to subsequently inform residents of the precise nature of their mistakes. And residents could theoretically perform well on the OSATS checklist even if they made an egregious error because points are not deducted, only earned in the grading system. To mitigate this, the researchers propose adding safety steps, such as identifying and protecting important structures, to the checklist.”

Dr. LaPorte told OTW, “What works well as far as tracking errors is to identify and document errors in real time. During a training session, errors should be identified as they occur and should be documented and discussed with the trainee. The trainee will then have a heightened awareness to avoid similar errors in the future.

“What does not work so well is trying to recall errors at the end of a day or at the end of a teaching session. Self-evaluation also is not ideal for tracking errors as the trainee may not always recognize that they have committed an error.

“We were pleasantly surprised to learn that this combination of multiple evaluations is feasible, reliable, and valid and can be applied to multiple different orthopedic procedures.”