EXAMINING FEEDBACK LOOP THINKING AS A MENTAL MODEL: DEVELOPMENT AND ANALYSIS OF A NEW MEASURE

Abstract

This study investigates feedback loop thinking and novices' skills in identifying feedback loops, aiming to create a valid assessment tool that distinguishes real understanding from mere awareness. The validity of the tool is assessed in two ways, 1) using it to test hypotheses about feedback learning by deploying the tool to test the effects of the terms used to name a feedback loop, and outcome desirability on loop identification accuracy determined by a signal detection analysis, and 2) using an item response analysis to determine the value of each item in distinguishing among students with a range of understanding of feedback loops. Results show little effect of terminology on sensitivity or bias, suggesting educators can use their field’s preferred terms without compromising outcomes, but a clear effect of the desirability of outcome on both sensitivity and bias, with a small increase in sensitivity and a clear bias to classify items as loops if they have a good outcome. The analysis of item sets for both positive and negative feedback loop using Item Response Theory (IRT) identified 16 items for each type of loop that provided a range of difficulties and discrimination over a moderately broad range of skill. Model fit favored a 2PL model, but global fit statistics were precluded by missing data. Still, this project delivers a reliable task for understanding feedback loop thinking, a crucial step for enabling future research on complex causal relationships. Subsequent studies should focus on assessing the unidimensionality of the item sets and elucidating the differences in how novices perceive positive versus negative feedback loops. Such efforts will further advance our understanding of feedback loop cognition.