How Effective Are Letters of Recommendation to Residency Programs?

To what extent do readers of those (largely) glowing letters of recommendation for medical students actually buy what the writers are saying? That’s what a team of researchers from Massachusetts General Hospital and Boston Medical Center set out to determine.

Their study, “Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs?” was published in the August 2021 edition of Clinical Orthopaedics and Related Research.

As someone who reads those letters each year, Paul Tornetta, III, M.D., chief and chair, professor and residency program director in the Department of Orthopaedic Surgery at Boston University School of Medicine, is definitely curious about the extent to which his perception of the letters matches the intent of the letter writer. Explaining that this is a “first” in the literature, Dr. Tornetta, a co-author on this work, told OTW, “The concordance of how a letter of recommendation is viewed compared to its intent has not been previously assessed.”

The researchers asked two questions: “1) Is the perception of letter readers for narrative letters of recommendation consistent with the intention of the letter’s author? (2) Is there inter-reviewer consistency in selection committee members’ perceptions of the narrative letters of recommendation?”

Individuals who wrote two or more letters of recommendation for applicants to one university-based orthopedic residency program for the 2014 to 2015 application cycle were asked to complete a survey associated with a specific letter of recommendation they wrote.

The team sent 247 unstructured letters of recommendation and the accompanying surveys to their authors, with 157 surveys returned (64% responded). “The seven core members of the admissions committee (of 22 total reviewers) at a university-based residency program were sent a similar survey regarding their perception of the letter,” wrote the study authors.”

“To answer our research question about whether letter readers’ perceptions about a candidate were consistent with the letter writer’s intention, we used kappa values to determine agreement for survey questions involving discrete variables and Spearman correlation coefficients to determine agreement for survey questions involving continuous variables. To answer our research question regarding inter-reviewer consistency among the seven faculty members, we compared the letter readers’ responses to each survey question using intraclass correlation coefficients.”

Dr. Tornetta summarized the study’s findings to OTW, “With the exception of the most senior member of the admissions committee, there was poor agreement between the belief of the letter writer in the candidates’ chances of success and the committee members assessment of the same. I believe that a sliding scale used in this article would be the most beneficial addition to the letter writing process.”