Winter 2016 Issue

Back to Home to Home Page         Print page PRINT PAGE


Messsge from the EditorEditor's Corner

Moneyball, FiveThirtyEight and Resident Applicant Ranking

By Robert G. Locke, DO, MPH, FACOP
Professor of Pediatrics/Neonatology

Robert G. Locke, DO, MPH, FACOP
Robert G. Locke, DO, MPH, FACOP

Over the next couple of weeks, our expanding neonatology practice is going to select four new attending neonatologists and participate in ranking fellowship (PGY4) candidates from hundreds of applicants. Everyone in our faculty practice has participated in residency ranking and medical student selection at some point. When scouring potential medical student applicants for interviews, I found that all potential candidates in my stack were smart, had been to Africa for a few weeks, and had received the same coaching on how best to present themselves and what to write in a personal statement. It was virtually impossible to distinguish one candidate from another.

Resident applications, on the other hand, are easier to review. Many of the applicants have rotated through the health system. Recent and relevant accomplishments with personalized letters of recommendation provide key insights. Even easier is the ranking of subspecialty fellow candidates. The smaller pool of applicants, the more specific focus of required skills, the ability to interview with greater attention, the attributes of the candidate are further distilled during residency. Personal letters of recommendations, especially when many of us know each other within a subspecialty field, carry meaningful weight.

After twenty years of reviewing resident applicants, I would like to tell you that I have it down to science, that an algorithm lets us accurately distinguish future performance of fellows/residents we select and personality melding and life-long productivity of fellow attendings. That would be a lie. Although resident/fellowship programs scrutinize the objective data, investing in selecting and training a resident, fellow or fellow attending still relies on fair amount of arbitrariness, bias and the all-important personal connection.

There is a fair amount written on how resident candidates need to present themselves to get selected, but there is little written on how residency programs actually select candidates, especially in pediatrics. A study of Urology Residency directors cited very low board scores as a detriment; while minority status, gender underrepresentation and couple match were factors of consideration, they were far less important than being a child of an academic urologist, completing a rotation at the program to which the application was sent and graduating from the same medical school as the program director. While this insider “old boys” network is something to overcome, not an excuse, the lack of good guidance for selection criteria creates a selection bias towards the familiar.

Studies have demonstrated that faculty characteristics can affect interview scores. Individual faculty characteristics and the “luck of the draw” can meaningfully influence an applicant’s rank, as much as 30 out of 150 places. It is likely that similar effect on residency ranking and clerkship experience are highly dependent upon the faculty member exposure, faculty member’s ability to write a positive letter and grading scale. I am a notorious easy grader. A colleague of mine and I often have the exact same assessment of a fellow candidate in the interview, but he will give them a six out of ten and I will score them a nine out of ten. Obviously, biased trouble or good news if only one of us gets to interview. For that reason, we and others have abandoned a simple ranking system, but there is no clear best method to perform the necessary ranking.

Beyond the simple test scores, what is the residency selection committee looking for? One residency director states that the ideal resident should have many of the following: great clinical knowledge; self-awareness and commitment to improvement; service orientation; pride in the profession; patient’s best interest in heart; creativity and innovation; conscience and trustworthiness; time management and prioritization; ethically sound decision making and leadership. A few residents wrote that a great resident should be: trustworthy (single most important characteristic); efficient; self-directed learner; detail-oriented; professional; personable; and academic.

I recently read The Signal and The Noise by Nate Silver, famed author of the FiveThirtyEight blog and one of the early originators of using stats and data to improve predictive analytics when selecting major league baseball players from their minor league performances. I was struck by the similarities between residency/fellowship selection and determining which minor league players to bring to the majors. Hard data is important. Scouting is important. Combining data with an excellent scout, then the predictive ability is increased above data alone. One top scout has a list of five tools that he looks for in a candidate. These are not dissimilar to what I believe are resident/fellow/attending characteristics that are associated with a happy and successful career: (1) intellectual curiosity and critical thinking; (2) preparedness and work ethic; (3) adaptiveness and learning ability; (4) concentration and focus; (5) competitiveness and self-confidence; (6) stress management and humility; (7) love of the work; (8) excellent interpersonal skills and (9) calmness and efficiency in a crisis.

At the fellow and attending level, board scores become less helpful. It is these other more crucial skills underpin differential and sustained success. In the near future, maybe predictive analytics will have an algorithm to measure intellectual curiosity, interpersonal flexibility, focus and work ethic while minimizing faculty interviewer and program director’s unsound or capricious biases.

In the meantime, I will put on my scouting hat on and do my subjective best.

Three Residents’ Perspectives on What Makes a “Great Resident”

Adopted from Venu Nemani, Caroline Park, Danyal Nawabi

One Program Director’s Perspective of an Ideal Resident

Adopted from Marta Miyares

One Faculty Member’s Perspective of Applicant Characteristics Associated
with a Future Success and Career Satisfaction

Back to top




member app







ACOP 2017 Spring Conference


ACOP 2017 Spring Conference

April 6-9, 2017
Hyatt Regency Louisville

Louisville, Kentucky