A Hierarchy of Grammatical Difficulty for Japanese EFL Learners: Multiple-Choice Items and Processability Theory

Abstract

This study investigated the difficulty order of 38 grammar structures obtained from an analysis of multiple-choice items using a Rasch analysis. The order was compared with the order predicted by processability theory and the order in which the structures appear in junior and senior high school textbooks in Japan. Because processability theory is based on natural speech data, a sentence repetition test was also conducted in order to compare the result with the order obtained from the multiple-choice tests and the order predicted by processability theory. The participants were 872 Japanese university students, whose TOEIC scores ranged from 200 to 875. The difficulty order of the 38 structures was displayed according to their Rasch difficulty estimates: The most difficult structure was subjunctive and the easiest one was present perfect with since in the sentence. The order was not in accord with the order predicted by processability theory, and the difficulty order derived from the sentence repetition test was not accounted for by processability theory either. In other words, the results suggest that processability theory only accounts for natural speech data, and not elicited data. Although the order derived from the repetition test differed from the order derived from the written tests, they correlated strongly when the repetition test used ungrammatical sentences. This study tentatively concluded that the students could have used their implicit knowledge when answering the written tests, but it is also possible that students used their explicit knowledge when correcting ungrammatical sentences in the repetition test. The difficulty order of grammatical structures derived from this study was not in accord with the order in which the structures appear in junior and senior high school textbooks in Japan. Their correlation was extremely low, which suggests that there is no empirical basis for textbook makers'/writers' policy regarding the ordering of grammar items. This study also demonstrated the difficulty of writing items testing the knowledge of the same grammar point that show similar Rasch difficulty estimates. Even though the vocabulary and the sentence positions were carefully controlled and the two items looked parallel to teachers, they often displayed very different difficulty estimates. A questionnaire was administered concerning such items, and the students' responses suggested that they seemed to look at the items differently than teachers and what they notice and how they interpret what they notice strongly influences item difficulty. Teachers or test-writers should be aware that it is difficult to write items that produce similar difficulty estimates and their own intuition or experience might not be the best guide for writing effective grammar test items. It is recommended to pilot test items to get statistical information about item functioning and qualitative data from students using a think-aloud protocol, interviews, or a questionnaire.