The Impact of Examinee Performance Information on Judges' Cut Scores in Modified-Angoff Standard Setting Exercises

Abstract

Background: Providing examinee performance data is a common procedural modification in modified-Angoff standard setting exercises. Results of research examining the impact of the practice are mixed, and despite the frequency with which this procedural modification is done, the overall impact on the outcomes is not well understood. This research presents a large-scale evaluation of the impact of providing performance data in the context of operational standard setting exercises for a medical licensing examination program. Methods: Data from a total of 18 independent standard setting panels across three different examinations were analyzed to evaluate whether and how the provision of performance information impacted the resulting cut scores. Results: Significant cut-score changes were found between initial and final judgments for all panels. A decrease in standard deviation between initial and final judgments for all panels was indicative of the convergence of judgments following the provision of data, and a lack of systematic changes in final cut scores suggested that judges were not simply changing their content-based judgments to align with the normative data. Both findings provide important procedural validity evidence for the standard setting process. Implications: The current data set (replications of the procedure within a single cycle, across cycles for an individual examination, and across multiple examinations) allowed for the most comprehensive evaluation of this topic that has been described to date and provides valuable insight into an issue that is relevant to many testing contexts.