Effects of L2 Affective Factors on Self-Assessment of Speaking

Abstract

This study was an investigation of the validity of students' self-assessment of L2 oral performance, the influences of L2 affective variables on their self-assessment bias, and the degree to which the influences of L2 affective variables differ between high and low proficiency learners. The participants were 389 science majors from two private Japanese universities. A questionnaire was administered using items based on the Attitude/Motivational Test Battery (Gardner, 1985), the Foreign Language Classroom Anxiety Scale (Horwtiz et al., 1986), the Rosenberg Self-Esteem Scale (Rosenberg, 1965), Sick and Nagasaka's (2000) Willingness to Communicate Scale, and items designed to measure motivation adapted from Gardner, Tremblay, and Masgoret (1997), Yashima (2002), Irie (2005), and Matsuoka (2006). A factor analysis identified seven factors in the questionnaire data: Self-Esteem, L2 Speaking Anxiety, L2 Willingness to Communicate, Attitude Toward Learning to Speak English, L2 Speaking Motivational Intensity, Desire to Learn to Speak English, and L2 Speaking Self-Confidence. The scales were further validated using the Rasch rating scale model. Student oral interviews were recorded and rated by five English teachers using an oral assessment scale based on the Kanda English Proficiency Test (Bonk & Ockey, 2003).Immediately after the interviews were completed, the participants were asked to consider how they perceived their own speaking performance and they rated their own performance from their memory using the same oral assessment scale that the teacher raters used. The oral assessment scale included the descriptions of the oral performances that match each level. The participants read the descriptions of each level and chose a level that they thought matched their own performance. The study produced four main findings. First, a multi-faceted Rasch analysis revealed that the participants rated their own L2 speaking more severely than the teacher raters and that the students' self-assessments were neither reliable nor consistent. Second, self-assessment bias measures were calculated and used to test a hypothesized structural model of how affective factors influenced self-assessment bias. The hypothesized model showed poor fit to the data, possibly due to the poor reliability of the self-assessment measures. Multiple regression analyses conducted as a follow-up analysis revealed that participants with greater Desire to Learn to Speak English tended to underestimate and those with greater L2 Speaking Self-Confidence tended to overestimate their own speaking performance. Third, 106 participants whose self-ratings were similar to the teachers' ratings were compared with other students in order to examine their distinctive features. However, no significant differences in L2 oral proficiency or affective variable measures were found between the two. Therefore, those whose self-assessments agreed with teachers could have resulted in some agreements that occurred by chance alone. Finally, 100 higher proficiency students were compared with 100 lower proficiency students and the results showed that the higher proficiency students with greater Desire to Learn to Speak English generally underestimated their L2 speaking proficiency, while those with higher Self-Esteem and greater L2 Speaking Self-Confidence tended to overestimate it. Lower proficiency students with greater L2 Speaking Self-Confidence tended to overestimate their L2 speaking proficiency. The results suggest that the self-assessment of L2 speaking might not be a sufficiently reliable or consistent assessment tool. Therefore, if teachers are considering including self-assessment in a speaking class, self-assessment training should be conducted. Additionally, giving L2 learners more opportunities to speak the L2 can help them notice gaps between their productions and those of proficient speakers, which might lead to more accurate self-assessment. Second, although some studies utilized only one teacher-rater, five teacher raters in this study displayed a great deal of diversity and exhibited unique bias patterns, so multiple raters should be employed and Facets analyses should be employed because the multi-faceted Rasch model provides person ability estimates that are adjusted for rater bias. Finally, the use of multi-faceted Rasch analysis is useful for examining oral data because unlike raw scores, multi-faceted Rasch analysis provides detailed information concerning speaker ability, rater severity, and category difficulty. Moreover, while most researchers have utilized self-assessment raw scores, in this study bias measures of self-assessment were calculated using Facets, which indicated that the bias measures produced different outcomes compared with self-assessment scores.