Key Study Guides Residency Selection

Just when you’ve got your “formula” down for selecting successful residents all of a sudden comes along a puzzler…someone who aced the boards, did impressive extracurricular research, and comes with excellent recommendations. But, it is becoming clear that he or she has difficulty interacting with others and is causing disruption in the program.

Is there a way to see this coming?

Dr. Kenneth Egol, vice chair for education in orthopaedic surgery at the New York University (NYU) Langone Medical Center is an author of a study entitled, “Success in Orthopaedic Training: Resident Selection and Predictors of Quality Performance.” For 20 years, NYU has collected data on its orthopedic residents…and, because the program is the largest orthopedic residency training program in the country—12 individuals a year—there is a rich store of data.

With the shifting sands in medical education, says Dr. Egol, it is more important than ever to get residency selection right. “It has always been vital to try and attract the best and the brightest. Nowadays, however, it is even more critical because of things such as work hour restrictions, issues with the Accreditation Council for Graduate Medical Education, etc.…these issues are changing the way we train people. In our study we set out to determine if there is a way to predict who will be successful.”

The team took a measured approach, says Dr. Egol: “We decided on what we considered to be good and bad performance; but, just because we associated certain factors with good or bad performance in the study we make no claim that this shows how they will do in practice.”

What You Can and Can’t Control

As the team proceeded with its retrospective review, they came to see that things were breaking down along the lines of “what we can control/know, and what we can’t.” “Each year we receive 700 applications for 12 residency places; in the end, we conduct thorough interviews with 72 people. No matter what precautions we take, however, every once in a while we find that we didn’t make the right choice.”

It really gives us pause…how is that possible if the person did well on boards, participated in extracurricular, research, etc.? We found that if there is a problem it’s typically in the areas we can’t control, namely, the affective domain (personality, etc.).

So despite doing well on orthopedic rotations, being a member of the Alpha Omega Alpha Honor Medical Society, etc…a troubled resident could still end up sitting in your office for a serious heart to heart. “There could be a work ethic issue, or the person could have problems with interpersonal relationships. There is usually a pattern of behaviors that they come with, so to speak, but this is not something that is easy to assess.”

Speaking of more concrete measures, Dr. Egol notes, “Our results indicated that when looking solely at academic pre-residency selection factors such as medical school rank and scores on part one of the standardized licensing exam, there was a positive correlation with higher scores on the Orthopaedic In-training Exam (OITE). And, although people like to downplay standardized tests, our study showed that the OITE is indeed a predictor of performance in our program and whether or not someone will pass the boards (the ultimate goal of completing training). However, this doesn’t mean that if someone does poorly on the OITE that he or she can’t pass the boards.”

“Other factors that correlated with success in residency included successful completion of Part I of the American Board of Orthopaedic Surgery (ABOS) certification exam on the first attempt, a mean clinical performance score based on evaluations following each clinical rotation during the residency, the number of peer-reviewed articles the resident had published during residency training, and being named an executive chief resident director. Regarding the last of these, the resident was selected on the criteria that he or she must be considered a role model for his or her peers, possess a good fund of knowledge, have leadership skills, and be adept at multitasking.”

Leadership and Time Management

When asked what particular leadership skills one might look for when interviewing candidates, and how they might be assessed, Dr. Egol is frank: “This is pretty difficult. The best we can do at this point is to try to evaluate the potential resident’s history of leadership roles within the organizations and activities they have participated in.”

The ability to handle a substantial workload cannot be emphasized enough when it comes to successful residents, says Dr. Egol. “The ‘problem’ with residency is that the person is still a student, yet they have a full time job…unlike when they were in medical school. The issue of being able to budget one’s time is key; people work at different speeds, of course, and when they have a lot of patients, consults, paperwork, etc., a slower worker or one who is not supremely organized affects the rest of the team because the work is left for them. We sometimes see that people make the mistake of thinking, ‘I worked like a dog for years to get here…now I can slow down a little.’ They are disconcerted to learn that residency is actually the time when they need to work harder than ever.”

George Washington University Medical SchoolThe NYU team also learned that those potential residents who extend themselves in non-medical directions are more likely to have a successful residency experience. “We found that well-rounded individuals such as those who do substantial charity work and participate in varsity athletics did better in residency. Specifically, those residents who were former varsity athletes received higher retrospective faculty evaluations on knowledge and were more likely to be named executive chief resident than were those who had not been collegiate varsity athletes. Given that success in residency involves leadership skills, team skills, a sense of individual responsibility, accountability, and time management skills—the things normally present in varsity athletes—it is perhaps no surprise that these individuals did better.”

Overall, these characteristics fall under the rubric of comportment. Dr. Egol notes, “Professionalism is an area where some residents could use a bit of guidance. Does the person interact well with others? Does he or she respond appropriately to situations? Much of this falls within the realm of communications. To this end, we have on faculty an American Academy of Orthopaedic Surgeons communications instructor who, in part, works to help residents understand the subtext of their communications. This fall we are actually piloting a thorough professionalism training program within our residency.”

Advice From the Experts

In conjunction with their efforts, states Dr. Egol, they tapped into the knowledge and experience of those who assess people for employment every day…the NYU human resources team. “We sat down with the people from HR to see if we could get to the bottom things. They helped us develop several questions that would help show whether potential residents possessed the ability to work as a team member, whether the person had a healthy set of interpersonal skills, etc. For example, ‘Tell me about an experience you had during medical school where you had to lead a team?’ or, ‘Talk about a time when you had to deal with a difficult patient.’ We also worked with them to develop several ethics-related scenarios such as, ‘What would you do if you were the junior resident on a service and felt that the attending was somehow impaired?’ While nine out of ten students said that they would approach the chief resident with his concerns, the fact is that they know this is the right answer, and there is always a chance that they would not actually proceed in this manner.”

So what does the future hold for the educational pioneers at NYU?

This year we will begin using the Defining Issues Test, a moral reasoning exam that may help us assess how potential residents would approach problems and/or deal with patients.

“The NYU medical school has been administering this to medical students for several years with some success. We hope that it will help us make progress in assessing residents in the affective domain.”

With recent advances in metrics, Dr. Egol and his team can be even more accurate in their assessments of potential residents. “We are using actors who simulate patients with orthopedic disease. The residents see these actors knowingly and unknowingly throughout their training. At various points in the program these ‘simulated patients’ give us feedback regarding the residents performance.”

“Going forward, ” says Dr. Egol, “we need a broad-scale, multicenter study to examine whether these findings are reproducible in other orthopedic surgery residency programs throughout the U.S. That way, we can more readily pinpoint the precise reasons for success in training.”