Associations between Doximity internal medicine residency navigator reputation rank and publicly available metrics
Graduate medical education (GME) has been scrutinized for lack of transparency and accountability in training the next generation of physicians . In 2014, the Institute of Medicine recommended that the $15 billion in federal funding provided for GME should align with the nation’s health care needs . This would include supporting and advancing the quality of United States GME training programs. Currently, no consensus exists regarding what constitutes markers of quality in residency training. Some have suggested that quality markers include patient outcomes, faculty characteristics, institutional support, and/or process measures . Therefore, evaluating the performance of GME training programs will require reliable methods. Since 2014, U.S. News & World Report has utilized the Doximity physician social networking site to determine reputation rankings for 27 specialties in the United States. The reputation rankings are identified through a survey of Doximity members who must register prior to being surveyed. Each year, physician members can vote for five residency programs within their specialty. Published studies suggest that medical students commonly read the Doximity residency reputation rankings, feel that they are moderately accurate, and even modify their rank lists based on these rankings. Since the publicly available Doximity reputation rankings may influence medical students' residency choices, further investigation of their associations with residency characteristics is warranted. The methodology for determining Doximity residency reputation rankings has been criticized as erroneous and unreliable. The Alliance for Academic Internal Medicine has warned that Doximity’s survey methodology is flawed and discouraged its use. Nonetheless, residency programs that do not encourage their faculty and alumni to participate in the survey could receive lower reputation rankings, which could negatively affect recruitment of outstanding medical student applicants. We aim to add to the existing literature and explore the associations between the Doximity Residency Navigator reputation rankings and objective residency program characteristics in internal medicine residency using data from a large variety of publicly available sources. Therefore, we conducted a cross-sectional study linking Doximity internal medicine residency reputation rankings and publicly available residency program metrics. The aim of this study was to determine whether Doximity internal medicine residency reputation rankings are associated with objective, publicly available data on residency programs. 

We conducted a cross-sectional study of all 566 internal medicine residencies with a program reputation rank assigned by the Doximity Residency Navigator in 2020. The residency program reputation ranking was linked with publicly available data sources including the American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online (FREIDA), the American Board of Internal Medicine (ABIM), the Accreditation Council for Graduate Medical Education (ACGME), the Centers for Medicare and Medicaid Services Five-Star Quality Rating System (CMS), and the U.S. News Best Hospitals Ranking Score (US News). When combining the data, all data was from 2020 (including ABIM pass rate), and the primary training hospital data was used for hospital level data. Resident, faculty, and program variables were selected a priori from all publicly available variables in these databases through consensus among all the study authors. Resident variables included international medical graduates (percent), Doctor of Osteopathy (percent), female (percent), and ABIM program pass rate (3-year average, percent). Faculty variables included physician faculty-to-resident ratio, female physician faculty (percent), non-physician faculty-to-resident ratio. Program variables included region (northeast, south, midwest, west), type (university-based, community-based, military, community-based university-affiliated), year founded, accreditation status (continued, continued with warning, initial, initial with warning, probationary, voluntary withdrawal), Centers for Medicare and Medicaid Services Star Rating (1–5, where 5 is higher), research rotation (required, optional, not offered), total approved resident positions, number of training tracks, number of training sites, time at primary training site (percent), U. S. News Primary Hospital Ranking Points (where more points is higher). This study was deemed exempt by the Mayo Clinic Institutional Review Board. Statistical analysis Categorical variables were summarized as numbers and percentages. Continuous variables were presented as means and standard deviations (SD). Associations between the independent variables of resident, faculty, and program characteristics and the dependent variable of Doximity residency reputation rank, were first examined using univariable linear regression. Regression coefficients were presented as β and standard error (SE). Variables having a p -value ≤ 0.05 in univariable analysis were included in the multivariable linear regression model. Data analyses were conducted using SAS software version 9.4 (SAS Institute, Cary, NC). 

Categorical variables were summarized as numbers and percentages. Continuous variables were presented as means and standard deviations (SD). Associations between the independent variables of resident, faculty, and program characteristics and the dependent variable of Doximity residency reputation rank, were first examined using univariable linear regression. Regression coefficients were presented as β and standard error (SE). Variables having a p -value ≤ 0.05 in univariable analysis were included in the multivariable linear regression model. Data analyses were conducted using SAS software version 9.4 (SAS Institute, Cary, NC). 

In the univariable analysis, all publicly available variables, except for percent female physician faculty (−0.62 ± 0.40, P = 0.127) and percent time spent at primary training site (−0.11 ± 0.32, 0.718), were significantly associated with program rankings ( p ≤ 0.05), with non-physician faculty-to-resident ratio and physician faculty-to-resident ratios having the strongest parameter estimates. All publicly available characteristics that were statistically significant were included in the multivariable model and are summarized in Table . Not all variables of interest were reported by all residency programs (reporting rate percent for individual variables = 85% to 100%). The availability of a research rotation variable was dichotomized to yes or no for the multivariable analysis. The multivariable analysis (Table ) revealed (parameter estimates ± standard error in parentheses) that a higher ABIM program pass rate (−1.30 ± 0.41), higher physician faculty-to-resident ratio (−14.10 ± 4.45), greater years since program founding (−1.07 ± 0.25), availability of a research rotation (−38.56 ± 16.23), and larger program size (−1.09 ± 0.16) were associated with a higher program Doximity rank. A higher percentage of international medical graduates (1.07 ± 0.19) and Doctors of Osteopathy (1.12 ± 0.27) were associated with a lower program Doximity rank. For these seven variables, the adjusted R 2 was 0.53. 

This study revealed that Doximity residency reputation rankings are positively associated with objective, publicly available characteristics of residency programs including higher physician faculty-to-resident ratio, larger program size, older program age, availability of a research rotation, and a higher three-year rolling ABIM pass rate. Doximity rankings were negatively associated with higher percentages of residents who were IMG or DO. Overall, these results align with previous studies on markers of residency program quality [ , , ]. We found that Doximity reputation rankings strongly correlated with the availability of research rotations. Publishing and research productivity are known to impact university rankings in other academic fields, and in non-internal medicine specialties within the field of Medicine. Additionally, resident research productivity has become an extremely important metric for resident selection into academic faculty positions and competitive fellowship programs., Finally, research productivity brings visibility to residency programs, which would tend to increase a program’s reputation. All covariates in the multivariable model were statistically significant, underscoring the independent contributions of each variable to the Doximity rankings. This reinforces the model's robustness in explaining the variance in rankings (adjusted R2 = 0.53). Doximity rankings also strongly correlated with program size and high faculty-to-resident ratios. Higher faculty-to-resident ratios would tend to enhance the quality of resident education may provide greater research mentoring opportunities and may be perceived as more appealing. Moreover, programs with more faculty members may be more likely to advance their reputations through the dissemination of research and establishing recognition for scholarly expertise. Although we are uncertain what percentage of individuals vote for their own program, this outcome may provide larger residency programs an advantage in the Doximity rankings. Other factors, such as ABIM board pass rate and years since program founding, lead to more favorable Doximity rankings, though these associations were more modest. More established residency programs may have a larger pool of alumni to vote for their own programs. It is anticipated that ABIM board pass rates would be associated with favorable Doximity rankings. However, US medical graduates generally have higher pass rates than IMGs , and we identified a negative association between Doximity rankings and programs with more IMGs; our datasets did not allow further analysis for these variables, and it remains uncertain the extent to which IMG status may overlap with lower ABIM pass rates for some programs. Finally, we would observe that community-based internal medicine residency programs are more likely to have IMGs and DOs, and lower peer-reviewed research productivity  than their academic counterparts. This, along with community programs’ small size, would tend to put them at a disadvantage in the Doximity reputation rankings. Additionally, academic programs are often considered more prestigious due to access to fellowship opportunities and external funding, which reinforce their reputation. External funding and research grant awards likely contribute to the visibility and prestige of these programs, which may further enhance a program’s perceived reputation. This was not directly examined in this study but merits further investigation. Our research findings have multiple implications. First, the Doximity rankings appear to emphasize research productivity, which is essential for advancing residents into competitive positions and is crucial for furthering medical science. Second, community-based programs may be disadvantaged by the Doximity rankings, due to their smaller sizes, fewer research opportunities, and higher proportions of IMG and DO students, all of which were variables that correlated negatively with Doximity rankings in this study. It is worth emphasizing that community-based internal medicine programs serve vital roles for the nation’s healthcare and education of future physicians. Community-based programs tend to produce the most primary care physicians; with the projected primary care shortage, strategies to increase the number of trainees entering primary care is crucial . Third, larger programs had higher Doximity reputations, which indicates their disproportionate representation in the Doximity voting methodology, and which supports prior research in other specialties showing associations between program size and Doximity rankings . Finally, our study provides insight into areas, like improving research opportunities and faculty-to-resident ratios, that may help program directors identify areas to improve their programs’ rankings. Study strengths and limitations This study has several strengths. First, it includes a large sample size and employs multivariable analysis to examine the relationship between residency rankings and program characteristics. Second, we assessed both academic and community-based internal medicine residency programs, enhancing the generalizability of our findings across different program types. Third, while some data points were missing across programs, our study utilized a robust national dataset, with missing values affecting no more than 15% of the 566 programs analyzed. However, our study also has limitations. Some residency programs did not report all variables of interest, leading to missing data and potentially overrepresenting programs with more comprehensive reporting. Additionally, we focused solely on internal medicine residency programs, limiting the applicability of our findings to other specialties. Furthermore, our dataset, which was based on publicly available information, did not include other potential indicators of program quality, such as patient outcomes, faculty characteristics, or resident research productivity. 

This study has several strengths. First, it includes a large sample size and employs multivariable analysis to examine the relationship between residency rankings and program characteristics. Second, we assessed both academic and community-based internal medicine residency programs, enhancing the generalizability of our findings across different program types. Third, while some data points were missing across programs, our study utilized a robust national dataset, with missing values affecting no more than 15% of the 566 programs analyzed. However, our study also has limitations. Some residency programs did not report all variables of interest, leading to missing data and potentially overrepresenting programs with more comprehensive reporting. Additionally, we focused solely on internal medicine residency programs, limiting the applicability of our findings to other specialties. Furthermore, our dataset, which was based on publicly available information, did not include other potential indicators of program quality, such as patient outcomes, faculty characteristics, or resident research productivity. 

To our knowledge, this is the first study to use national, publicly available datasets to demonstrate positive associations between Doximity reputation rankings and program size, program age, ABIM pass rates, faculty-to-resident ratios, and research elective opportunities. While it is certainly likely that the Doximity reputation rankings are inflated by voting from larger programs, many of the associations identified in this study – such as increased faculty ratios, research opportunities, and ABIM pass rates – suggest that the Doximity reputation rankings reflect important and meaningful characteristics of residency programs. This study should provide useful information to residency programs looking to more objectively describe metrics of program quality. 

