Rachel Yudkowsky

Competency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace “units of assessment”. Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents’ performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents’ performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other vari...

Background The Core Physical Exam (CPE) has been proposed as a set of key physical exam (PE) items for teaching and assessing PE skills in medical students, and as the basis of a Core + Cluster curriculum. Beyond the initial development of the CPE and proposal of the CPE and the Core + Cluster curriculum, no additional validity evidence has been presented for use of the CPE to teach or assess PE skills of medical students. As a result, a modified version of the CPE was developed by faculty at the University of Colorado School of Medicine (UCSOM) and implemented in the school’s clinical skills course in the context of an evolving Core + Cluster curriculum. Methods Validity evidence for the 25-item University of Colorado School of Medicine (UCSOM) CPE was analyzed using longitudinal assessment data from 366 medical students (Classes of 2019 and 2020), obtained from September 2015 through December 2019. Using Messick's unified validity framework, validity evidence specific to conte...

Supplemental Digital Content is available in the text. Purpose With the growing importance of professionalism in medical education, it is imperative to develop professionalism assessments that demonstrate robust validity evidence. The Professionalism Mini-Evaluation Exercise (P-MEX) is an assessment that has demonstrated validity evidence in the authentic clinical setting. Identifying the factorial structure of professionalism assessments determines professionalism constructs that can be used to provide diagnostic and actionable feedback. This study examines validity evidence for the P-MEX, a focused and standardized assessment of professionalism, in a simulated patient setting. Method The P-MEX was administered to 275 pediatric residency applicants as part of a 3-station standardized patient encounter, pooling data over an 8-year period (2012 to 2019 residency admission years). Reliability and construct validity for the P-MEX were evaluated using Cronbach’s alpha, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). Results Cronbach’s alpha for the P-MEX was 0.91. The EFA yielded 4 factors: doctor–patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. The CFA demonstrated good model fit with a root-mean-square error of approximation of .058 and a comparative fit index of .92, confirming the reproducibility of the 4-factor structure of professionalism. Conclusions The P-MEX demonstrates construct validity as an assessment of professionalism, with 4 underlying subdomains in doctor–patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. These results yield new confidence in providing diagnostic and actionable subscores within the P-MEX assessment. Educators may wish to integrate the P-MEX assessment into their professionalism curricula.

In the past few years, there has been rapid growth in the number of simulation fellowships for physicians in the United States and Canada, with the objective of producing faculty with expertise and leadership training in medical simulation. Relatively little is known about the collective content and structure of these new fellowship opportunities.Background  We sought to identify a common set of core curricular elements among existing simulation fellowships and to obtain demographic background information on participants and leadership.Objective  We designed a web-based survey and circulated it to simulation fellowship directors in the United States and Canada. The questions explored aspects of the fellowship curriculum. A grounded theory approach was used to qualitatively analyze fellowship goals and objectives.Methods  Of the 29 program directors surveyed, 23 responded (79%). The most commonly listed goals and objectives were to increase skills in simulation curriculum development...

Ventriculostomy is a neurosurgical procedure for providing therapeutic cerebrospinal fluid drainage. Complications may arise during repeated attempts at placing the catheter in the ventricle. We studied the impact of simulation-based practice with a library of virtual brains on neurosurgery residents' performance in simulated and live surgical ventriculostomies. Using computed tomographic scans of actual patients, we developed a library of 15 virtual brains for the ImmersiveTouch system, a head- and hand-tracked augmented reality and haptic simulator. The virtual brains represent a range of anatomies including normal, shifted, and compressed ventricles. Neurosurgery residents participated in individual simulator practice on the library of brains including visualizing the 3-dimensional location of the catheter within the brain immediately after each insertion. Performance of participants on novel brains in the simulator and during actual surgery before and after intervention was analyzed using generalized linear mixed models. Simulator cannulation success rates increased after intervention, and live procedure outcomes showed improvement in the rate of successful cannulation on the first pass. However, the incidence of deeper, contralateral (simulator) and third-ventricle (live) placements increased after intervention. Residents reported that simulations were realistic and helpful in improving procedural skills such as aiming the probe, sensing the pressure change when entering the ventricle, and estimating how far the catheter should be advanced within the ventricle. Simulator practice with a library of virtual brains representing a range of anatomies and difficulty levels may improve performance, potentially decreasing complications due to inexpert technique.

Importance The sociopolitical and cultural context of graduate surgical education has changed considerably over the past 2 decades. Although new structures of graduate surgical training programs have been developed in response and the comparative value of formats are continually debated, it remains unclear how different time-based structural paradigms are preparing trainees for independent practice after program completion. Objective To investigate the factors associated with trainees' and program directors' perception of trainee preparedness for independent surgical practice. Design, Setting, and Participants This qualitative study used an instrumental case study approach and obtained information through semistructured interviews, which were analyzed using open-and-focused coding. Participants were recent graduates and program directors of vascular surgery training programs in the United States. The 2 training paradigms analyzed were the integrated vascular surgery residenc...

The unexpected discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam in January 2021 carries both risks and opportunities for medical education in the United States. Step 2 CS had far reaching effects on medical school curricula and school-based clinical skills assessments. Absent the need to prepare students for this high-stakes exam, will the rigor of foundational clinical skills instruction and assessment remain a priority at medical schools? In this article, the authors consider the potential losses and gains from the elimination of Step 2 CS and explore opportunities to expand local summative assessments beyond the narrow bounds of Step 2 CS. The responsibility for implementing a rigorous and credible summative assessment of clinical skills that are critical for patient safety as medical students transition to residency now lies squarely with medical schools. Robust human simulation (standardized patient) programs, including regional and virtual simulation consortia, can provide infrastructure and expertise for innovative and creative local assessments to meet this need. Novel applications of human simulation and traditional formative assessment methods, such as workplace-based assessments and virtual patients, can contribute to defensible summative decisions about medical students' clinical skills. The need to establish validity evidence for decisions based on these novel assessment methods comprises a timely and relevant focus for medical education research.

To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs). Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores. Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlati...

Availability of reliable, valid, and feasible workplace-based assessment (WBA) tools is important to allow faculty to make important and complex judgments about resident competence. The Minicard is a WBA direct observation tool designed to provide formative feedback while supporting critical competency decisions. The purpose of this study was to collect validity and feasibility evidence for use of the Minicard for formative assessment of internal medicine residents. We conducted a retrospective cohort analysis of Minicard observations from 2005-2011 in 1 institution to obtain validity evidence, including content (settings, observation rates, independent raters); response process (rating distributions across the scale and ratings by month in the program); consequences (qualitative assessment of action plans); and feasibility (time to collect observations). Eighty faculty observers recorded 3715 observations of 73 residents in the inpatient ward (43%), clinic (39%), intensive care (15...

Health care errors are a national concern. Although considerable attention has been placed on reducing errors since a 2000 Institute of Medicine report, adverse events persist. The purpose of this pilot study was to evaluate the effect of mindfulness training, employing the standardized approach of an eight-week mindfulness-based, stress reduction program on reduction of nurse errors in simulated clinical scenarios. An experimental, pre- and post-test control group design was employed with 20 staff nurses and senior nursing students. Although not statistically significant, there were numerical differences in clinical performance scores from baseline when comparing mindfulness and control groups immediately following mindfulness training and after three months. A number of benefits of mindfulness training, such as improved listening skills, were identified. This pilot study supports the benefits of mindfulness training in improving nurse clinical performance and illustrates a novel a...

Communication and interpersonal skills (CIS) are essential elements of competency-based education. We examined defensible CIS passing levels for medical students completing basic sciences (second-year students) and clinical training (fourth-year students), using five standard setting methods. A 14-item CIS scale was used. Data from second-year (n = 190) and fourth-year (n = 170) students were analyzed using descriptive statistics and generalizability studies. Fifteen judges defined borderline CIS performance. Cut scores and fail rates from five standard setting methods (Angoff, Borderline-Group, Borderline-Regression, Contrasting-Groups, and Normative methods) were examined. CIS performance was similar during second-year (Mean = 74%, SD = 6%) and fourth-year (Mean = 72%, SD = 5%) students. Judges using the Angoff method expected greater competence at the fourth-year level, as reflected in the Angoff cut scores (second-year = 53% with 0% fail, fourth-year = 66% with 10% fail). Cut sc...

Construct: This study aims to provide validity evidence for the standardized Assessment Letter for Pediatrics as a measure of competencies expected of a 1st-year pediatrics resident as part of a pediatric residency admissions process. The Narrative Letter of Recommendation is a frequently used tool in the residency admissions process even though it has poor interrater reliability, lacks pertinent content, and does not correlate with residency performance. A newer tool, the Standardized Letter, has shown validity evidence for content and interrater reliability in other specialties. We sought to develop and provide validity evidence for the standardized Assessment Letter for Pediatrics. All 2012 and 2013 applicants invited to interview at the University of Geneva Pediatrics Residency Program provided 2 standardized Assessment Letters. Content for the letter was based on CanMEDS roles and ratings of 6 desired competencies and an overall assessment. Validity evidence was gathered for in...

Defensible minimum passing standards (MPSs) must be used to evaluate learner performance outcomes in health professions education. In this study, the authors compared the results of traditional Angoff and Hofstee standard-setting exercises with the Mastery Angoff and Patient-Safety approaches for central venous catheter (CVC) insertion skills examinations. The authors also evaluated how these standards affected evaluation of the historical performance of residents who participated in a simulation-based mastery learning (SBML) curriculum for CVC insertion skills. In April and May 2015, twelve physicians with expertise in CVC insertion set MPSs for previously published internal jugular (IJ) and subclavian (SC) CVC insertion checklists using Angoff, Hofstee, Mastery Angoff, and Patient-Safety approaches. The resulting MPSs were compared using historical performance of internal medicine and emergency medicine residents who participated in CVC insertion SBML. The MPSs were set as follows...

Second-year medical students performed a head-to-toe screening physical examination on trained patient instructors (PIs) as a high-stakes examination before beginning clinical clerkships. PIs completed a 138-item checklist and instructed the students in the proper performance of any incorrectly executed or omitted items. To assess the accuracy of the PI ratings, 'monitor' PIs watching the encounter by video completed checklists for 11 retests of failed students in 2001 and for 28 randomly selected encounters in 2002. Checklist-level discrepancies measured by mean absolute checklist difference were 8% in 2001 and 9% in 2002. Case-level mean difference between raters in 2002 was -2.9%. The inter-rater reliability for a single rater, estimated by the intra-class correlation coefficient for total encounter scores, was 0.95. The assessment and feedback were well received by the students. The Head-to-Toe examination provides a reliable assessment of a student's ability to carry out a complete screening physical examination. Patient instructors can be trained to acceptable accuracy and reliability despite the length and complexity of the exercise, and can provide effective feedback and teaching targeted to individual student deficiencies.

A contextual error occurs when a physician overlooks elements of a patient's environment or behavior that are essential to planning appropriate care. In contrast to biomedical errors, which are not patient-specific, contextual errors represent a failure to individualize care. To explore the frequency and circumstances under which physicians probe contextual and biomedical red flags and avoid treatment error by incorporating what they learn from these probes. An incomplete randomized block design in which unannounced, standardized patients visited 111 internal medicine attending physicians between April 2007 and April 2009 and presented variants of 4 scenarios. In all scenarios, patients presented both a contextual and a biomedical red flag. Responses to probing about flags varied in whether they revealed an underlying complicating biomedical or contextual factor (or both) that would lead to errors in management if overlooked. 14 practices, including 2 academic clinics, 2 community-based primary care networks with multiple sites, a core safety net provider, and 3 U.S. Department of Veterans Affairs facilities. Primary outcomes were the proportion of visits in which physicians probed for contextual and biomedical factors in response to hints or red flags and the proportion of visits that resulted in error-free treatment plans. Physicians probed fewer contextual red flags (51%) than biomedical red flags (63%). Probing for contextual or biomedical information in response to red flags was usually necessary but not sufficient for an error-free plan of care. Physicians provided error-free care in 73% of the uncomplicated encounters, 38% of the biomedically complicated encounters, 22% of the contextually complicated encounters, and 9% of the combined biomedically and contextually complicated encounters. Only 4 case scenarios were used. The study assessed physicians' propensity to make errors when every encounter provided an opportunity to do so and did not measure actual error rates that occur in primary care settings because of inattention to context. Inattention to contextual information, such as a patient's transportation needs, economic situation, or caretaker responsibilities, can lead to contextual error, which is not currently measured in assessments of physician performance. U.S. Department of Veterans Affairs Health Services Research and Development Service

Theoretical and empirical support is increasing for mastery learning, in which learners must demonstrate a minimum level of proficiency before completing a given educational unit. Mastery learning approaches aim for uniform achievement of key objectives by allowing learning time to vary and as such are a course-level analogue to broader competency-based curricular strategies. Sound assessment is the cornerstone of mastery learning systems, yet the nature of assessment validity and justification for mastery learning differs in important ways from standard assessment models. Specific validity issues include (1) the need for careful definition of what is meant by "mastery" in terms of learners' achievement or readiness to proceed, the expected retention of mastery over time, and the completeness of content mastery required in a particular unit; (2) validity threats associated with increased retesting; (3) the need for reliability estimates that account for the specific me...

The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane's formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon s...

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing...

ABSTRACT Purpose: A “contextual error” is a medical error that occurs when a physician fails to take into account information that is expressed outside of a patient´s physical boundaries – i.e. their context – that is essential to planning appropriate care. Using incognito standardized patients, we measured the propensity of physicians to elicit contextual (vs. biomedical) information and to make contextual (vs. biomedical) errors in treatment plans, as well as the avoidable direct costs of these errors to patient care. Method: Over 18 months, 98 internal medicine attending physicians at 8 Midwestern VA and non-VA practices were visited in vivo by incognito actors presenting variants of 4 previously-validated cases that jointly manipulated the presence or absence of contextual and biomedical factors that could lead to errors in treatment. For each visit, we obtained data on whether the physician elicited necessary information and incorporated it in the treatment plan recorded in the visit note. Mixed models were fitted to examine factors associated with failed elicitation or incorporation of information. Costs of missed services or unnecessary services in each visit were computed using Medicare cost-based reimbursement data. Result: Biomedical and contextual information were elicited equally often when available (70% vs. 69%). In baseline variants, treatment plans were appropriate in 76% of visits; when errors were possible, 29% of plans were appropriate when only a biomedical error was possible, 15% of plans were appropriate when only a contextual error was possible, and 2% of plans were appropriate when both types of errors (all differences significant). Most errors involved failure to order necessary services, but contextual errors were more likely to result in simultaneous underuse of necessary services and provision of unnecessary services. Contextual errors alone resulted in an estimated additional cost of care of $74,697 (median $194/visit), which was significantly greater than the $14,967 associated with biomedical errors alone (median $23/visit; Wilcoxon p<.001). Conclusion: Inattention to contextual information, such as patients´ transportation, economic situation, or caretaker responsibilities can have dramatic and measurable implications for quality and cost of care, in some cases beyond those resulting from inattention to laboratory values and medication dosages when delivering care. This study suggests a need for greater prioritization of contextual information in planning patients´ care to reduce medical errors and costs.

To determine the psychometric characteristics of diagnostic justification scores based on the patient note format of the United States Medical Licensing Examination Step 2 Clinical Skills exam, which requires students to document history and physical findings, differential diagnoses, diagnostic justification, and plan for immediate workup. End-of-third-year medical students at one institution wrote notes for five standardized patient cases in May 2013 (n = 180) and 2014 (n = 177). Each case was scored using a four-point rubric to rate each of the four note components. Descriptive statistics and item analyses were computed and a generalizability study done. Across cases, 10% to 48% provided no diagnostic justification or had several missing or incorrect links between history and physical findings and diagnoses. The average intercase correlation for justification scores ranged from 0.06 to 0.16; internal consistency reliability of justification scores (coefficient alpha across cases) ...

Guidelines for reporting several types of medical studies have been described in the literature. However, there are no current guidelines to report studies on mastery learning curriculum development and trainee evaluation in medical education. Such guidelines will be important because medical education is moving toward a competency-based model. The authors sought to define standards for the evaluation of mastery learning curricula using previously published guidelines in related fields and expert review.The authors reviewed previously published guidelines from clinical medicine, medical education, and the social sciences. Six authors with expertise in mastery learning curricula, performance assessment, and medical education compiled and reached agreement about a list of guidelines. The authors later circulated the list to 12 other experts and made revisions. A final list of guidelines was established and received group consensus. The Reporting Mastery Education Research in Medicine ...

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential.Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past e...