Coherence in Quantitative Longitudinal Language Program Evaluation

Abstract

In recent years, foreign language program evaluation has gained greater attention among language educators, program administrators, and evaluators. Increased demands for demonstrated program performance, often motivated by external forces, such as accreditation pressures and decisions regarding the allocation of funding, have led to heightened focus on foreign language program evaluation practices, methodologies, and results. Despite this increased attention, there are few published evaluation studies within the field of foreign language learning that have examined foreign language program effectiveness over time. This longitudinal study was designed to quantitatively investigate the performance of one Japanese university English for Academic Purposes (EAP) program over the 20-year span of the program’s existence. Quantitative evaluation methodologies and advanced statistical procedures were utilized to examine changes in student English proficiency, as measured by the Institutional Test of English as a Foreign Language (TOEFL ITP) and English achievement, as measured by four semesters of EAP course grades, as students progressed through the two-year program. Twenty cohorts of students (cohort n-sizes ranging from approximately 250 to 550 students) were included in this study. The comprehensive data set included three repeated-measures of the TOEFL ITP and four English achievement grade point averages (GPAs) for each of the 20 cohorts. The research questions for this expansive longitudinal study addressed two levels of inquiry. First, at the program-global level, this study sought to investigate patterns of English proficiency change within and between cohorts across the life of the program, and the extent that programmatic events and external influences might have impacted those patterns. For this investigation, TOEFL ITP results for three proficiency domains—listening, grammar, and reading—were chronologically charted for the 20 cohorts and time-series analyses were conducted. The results indicated that all cohorts demonstrated significant gains in the three proficiency domains by the end of the two-year program. However, the overall trends across the program’s 20-year history revealed gradual negative trajectories for grammar and reading proficiency. Events that were hypothesized to have influenced proficiency patterns were tested, including (a) the addition of a new department specialization, (b) changes to department admissions, (c) the entrance of students who experienced new national reforms at the secondary education level, and (d) department expansion. While listening proficiency patterns were unaffected, grammar and reading proficiency trends were negatively impacted by the start of the new specialization and changes to admissions procedures. The entrance of students who had experienced secondary educational changes had an initial negative impact on the grammar trend, but positive grammar and reading proficiency trends emerged from that point onward. It was speculated that these events, as well as larger population trends impacting Japanese universities, led to gradual shifts in program student demographics, which contributed to the observed changes in proficiency patterns. Also of interest was an examination of the concept of English achievement coherence—or the extent that student English achievement, as measured by English course grade point averages (GPAs)—can be used to assess course interrelatedness. English course GPA data was used to statistically derive three rival achievement coherence metrics. These metrics were then tested separately, using hierarchical linear modeling techniques, to examine the extent that achievement coherence might serve to mediate any proficiency variation observed across the 20 cohorts. There were no significant findings for two of the metrics tested, while the third metric was found to have a significant negative effect for reading proficiency. This finding directly contradicted the hypothesized outcome that a greater amount of coherence would serve to facilitate proficiency development. Given the significant negative reading trend that emerged across the life of the program, this result might suggest that larger influences affecting student demographic changes could outweigh any potential facilitative effects of coherence on proficiency outcomes. Following the program-global analyses, the second level of inquiry was at the cohort-specific level. Individual cohorts that had demonstrated comparatively high and low listening and reading proficiency gains were selected for follow-up analyses. The aim was to examine if differences in coherence at the cohort level might account for the contrastive proficiency gains attained. For each target cohort, a recursive path model, including the program’s 16 English courses and final proficiency outcome, was tested to examine English achievement interrelatedness and contributions to the final proficiency outcome. A greater number of significant paths and larger final model R2 coefficient would suggest more coherence. Additionally, for each target cohort, grade residuals analyses using linear regression methods were conducted to investigate grading consistency at the course level. A greater number of outlying grade cases could indicate that the course assessment schemes were not followed, which would suggest less cohort coherence. The results of these analyses for the pairs of contrastive listening gain and reading gain cohorts were compared, but no significant differences were found. While these analytical methods were determined to be useful for ongoing formative evaluation processes, the resulting measures were likely too broad to capture any meaningful differences in coherence between cohorts at the program-global level.