THE ASSOCIATION BETWEEN MEDICAL EDUCATION ACCREDITATION AND THE EXAMINATION PERFORMANCE OF INTERNATIONALLY EDUCATED PHYSICIANS SEEKING CERTIFICATION IN THE UNITED STATES

Abstract

Background: Physicians do not always provide appropriate patient care, due in part to inadequacy in their education and training. Performance outcomes, such as individuals' examination scores have been linked to future performance as physicians, accentuating the need for high-quality educational institutions. While the medical school accreditation process in the United States assures a uniform standard of quality, approximately one quarter of physicians in training and in practice in the United States graduated from medical schools located outside of the United States or Canada. These graduates of international medical schools (IMGs) have been more likely than domestically educated doctors to practice primary care and treat underserved and minority populations. An increasing proportion of IMGs who seek to enter post-graduate training programs and subsequent licensure in the United States graduated from medical schools located in the Caribbean. The quality of medical education at some of these schools has been questioned. Accreditation systems are frequently viewed as a way to ensure the quality of medical education, although currently there is limited data linking an educational oversight mechanism to better performance of the graduates. In addition, accreditation systems vary in the methodology, standards, and procedures used to evaluate educational programs. The purpose of the first phase of the present research was to examine medical education accreditation practices around the world, with special focus on the Caribbean region, to determine the association of accreditation of medical schools with student/graduate performance on examinations. The aim of the second phase of this research was to evaluate the quality of a select group of accrediting agencies and the association of quality with student/graduate outcomes. Methods All IMGs seeking to enter graduate training positions in the United States must first be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). In addition to other requirements, ECFMG certification includes passing scores on the United States Medical Licensing Examination (USMLE) Step 1 (basic science), Step 2 Clinical Knowledge (CK), and Step 2 Clinical Skills (CS). In the first phase, all IMGs taking one or more examinations leading to ECFMG certification during the five-year study period (January 1, 2006 through December 31, 2010), and who graduated from, or attended at the time of testing, a school located in a country that met the accreditation inclusion criteria, were included in the study population. First-attempt pass rates for each examination were calculated based on personal variables (gender, years elapsed since graduation at the time the individual took an examination [<3 years versus ≥ years], native language [English versus all others]), and on accreditation status of an individual's medical school. Next, separately for each examination, a generalized estimating equations model was used to investigate the effect of accreditation after controlling for the personal variables. Following the assessment of accreditation on test performance at the global level, the same analyses were conducted separately on the data from students/graduates who attended medical schools located in the Caribbean, and on the data from students/graduates who attended medical school not located in the Caribbean. In the second phase, the quality of a select group of accrediting agencies was evaluated according to the criteria determined by a panel of experts to be the most salient features of an accreditation system. Accreditation systems that used 80% or more of the criteria were given a quality grade of A, and systems using less than 80% of the criteria were given a grade of B. The association between the quality of an accreditation system and student performance, as measured by first-attempt pass rates on USMLE, was investigated in this second phase. The Temple University Office for Human Subject Protections Institutional Review Board determined by expedited review that this study qualified for exemption status. Results As of January 2011, there were 173 countries with medical schools listed in the International Medical Education Directory (IMED), of which 118 met the inclusion criteria. During the study period approximately 67,000 students/graduates took Step 1 for the first time, 55,600 took Step 2 CK, and 58,200 took Step 2 CS. Over one quarter of the test takers graduated from, or were students at, schools located in the Caribbean. For the global population, better performance on Step 1 was associated with the male gender, testing within three years of graduation, non-native English-speaking status, and attending a school located in a country with a system of accreditation. For the Caribbean population on Step 1, results were similar, except native English speakers outperformed non-native English speakers. After controlling for covariates, the odds of passing Step 1 for those from accredited schools were 1.8 times greater for the global group and 4.9 times greater for the Caribbean group as compared to the odds of passing the examination on the first attempt for individuals from nonaccredited schools. In contrast, in the non-Caribbean group accreditation was not associated with examination performance. Increased performance on Step 2 CK for the global group was associated with the female gender, testing within three years of graduation, non-native English-speaking status, and attending a school located in a country with a system of accreditation. For the Caribbean population on Step 2 CK, females, those testing closer to graduation, and native English speakers outperformed their counterparts. After controlling for covariates, the odds of passing Step 2 CK for those from accredited schools were 1.3 times greater for the global group and 2.3 times greater for the Caribbean group as compared to individuals from nonaccredited schools. Accreditation was not associated with examination performance for the non-Caribbean group. For all three groups (global, Caribbean, and non-Caribbean), better performance on Step 2 CS was associated with the female gender, testing within three years of graduation, native English- speaking status, and attending a school located in a country with a system of accreditation. After controlling for covariates, the odds of passing Step 2 CS for those from accredited schools were 1.3 times greater for the global group, 2.4 times greater for the Caribbean group, and 1.1 times greater for the non-Caribbean group compared to individuals from nonaccredited schools. In phase two, the expert panel unanimously agreed on 14 essential standards that should be required by accrediting agencies to ensure the quality of physicians. Of the accreditation systems in 18 countries that were analyzed for inclusion of the criteria, four systems, used in 10 countries, were given a grade of A (included 80% or more of the essential standards), and eight systems, used in eight countries, were given a grade of B (included less than 80% of the essential standards). The IMGs attending medical schools accredited by a system that received a grade of A performed better on Step 1 and Step 2 CS as compared to IMGs attending medical schools that are accredited by a system receiving a grade of B. For Step 2 CK, the results were reversed. Certain essential standards were associated with better performance for all three examinations. Discussion The purpose of this study was to investigate the USMLE performance of graduates of international medical schools who voluntarily seek ECFMG certification based on variables related to the accreditation of their medical education programs. In this study, for the self-selected population who took examinations during the study period, accreditation was associated with better performance in specific regions and for some examinations. Of the three examinations, the existence of a system of accreditation had the strongest association with Step 1 performance for the global and Caribbean groups. Many accreditation criteria are directly related to aspects of the preclinical phase of education. The association between accreditation and Step 2 CS was positive for all three groups of students/graduates, although systems of accreditation may have less direct impact on student performance on clinical examinations as students' experiences in the clinical phase are likely more varied. Of the three groups, the existence of accreditation systems had the greatest associated with examination performance in the Caribbean, an important finding considering the large numbers of IMGs educated in this region seeking ECFMG certification and ultimately treating U.S. patients. The quality of accrediting agencies, as determined by the number of essential elements utilized in the systems, was positively associated with performance for Step 1 and Step 2 CS, but not Step 2 CK. The finding supporting the importance of a high-quality accreditation system on Step 2 CS performance is important due to the purpose of this examination in evaluating a physician's skills in a real world setting. This study lends some support to the value of accreditation. Due to the substantial resources needed to design and implement accreditation processes, these results provide some positive evidence beyond face validity, especially in the Caribbean region, that quality assurance oversight of educational programs is associated with the production of more highly skilled physicians, which in turn should improve the health care of patients in the United States and around the world.