Ortho Residency Directors Push For Standardization

How extensive is the issue of rank inflation, or more bluntly, hype, in applications for orthopedic residency slots. While residency applications tend to attract superior applicants when compared to their peers applying for other subspecialties, this issue is real and a council of residency directors has decided to address it formally.

The Council of Orthopaedic Residency Directors (CORD) of the American Orthopaedic Association has developed a standardized letter of recommendation which was discussed in detail in a new paper, “Evaluating the Standardized Letter of Recommendation Form in Applicants to Orthopaedic Surgery Residency,” which appears in the October 1, 2020 edition of the Journal of the American Academy of Orthopaedic Surgeons.

The authors of this paper reviewed 1,137 standardized letters of recommendation (SLOR) for 513 applicants, which provided applicant scores from the United States Medical Licensing Examination (USMLE) Step 1, grades in clinical rotations, Alpha Omega Alpha status, and the number of publications.

Jay Lieberman, M.D., professor and chair of the Department of Orthopaedic Surgery at the Keck School of Medicine and professor of Biomedical Engineering at the Viterbi School of Engineering of USC, both at the University of Southern California, was part of this effort and, as he explained to OTW, there does appear to be a certain amount of candidate rank inflation.

“Dr. William Levine (Columbia University) and I noted that when reviewing the SLOR that a majority of applicants received a summative rank of either ranked to match or in the top third. In addition, in the sections on Qualifications for Orthopaedic Surgery and Global Assessment, a majority of candidates received a 90th percentile or greater.”

When 92% of applicants received a rank which either matched or was the top one-third of their rank list, rank inflation becomes a question. The authors also found that objective measures such as grades and USMLE Step 1 scores had a “very weak to nonexistent correlation with the summative rank.” Poor interrater reliability was also a problem.

OTW asked Dr. Lieberman about the issue of rank inflation and what can be done. He explained,“The ability to differentiate applicants is limited by the prevalence of inflation in the summative ranks. This is a difficult problem to resolve since most orthopaedic surgery applicants have superior qualifications compared to their peers applying for other subspecialties.”

“A guideline helping an evaluator to fill out the standardized letter of recommendation would be useful. Another potential improvement could be to add another category, ‘Top 20%,’ to further differentiate between the ranked to match group and the top one-third. The addition of this type of ranking must be validated.”

“Although the form states that the student is to be compared with other orthopaedic applicants, evaluators are clearly disregarding this instruction,” wrote the authors. “Although we cannot say for certain why this is occurring based on our data, we think that a primary driver of the rank inflation is the letter writers’ desire to advocate for their students…”

“The usefulness of the standardized letter of recommendation when evaluating an applicant is limited because of the high rankings observed for all questions,” stated Dr. Lieberman.