Measurement of Korean EFL college students' foreign language classroom speaking anxiety: evidence of psychometric properties and accuracy of a computerized adaptive test (CAT) with dichotomously scored items using a CAT simulation

Assessment of foreign language speaking anxiety is considered pertinent to assisting practitioners to reduce learners’ speaking anxiety. Computerized adaptive testing (CAT) of severity of speaking anxiety has potential advantages over conventional paper–and–pencil (P&P) tests in terms of efficiency, precision, adaptiveness and real–time severity monitoring. Item response theory (IRT) was applied to obtain item characteristics (i.e. anxiety severity and discrimination parameter) and develop an item pool that can be used for a computerized adaptive test to measure severity of speaking anxiety. Based on two–parameter logistic IRT model, the study analyzed responses to a newly constructed English speaking anxiety inventory from a sample of 949 Korean EFL undergraduate students. The study used Principal Component Analysis and DIMTEST in a confirmatory mode to account for construct validity and conducted a computer simulation of CAT to investigate whether a CAT or a P&P test can more accurately estimate test–takers’ severity levels of English speaking anxiety, conditional on their true severity of speaking anxiety. Examining construct validity, the results indicated that a principal component analysis (PCA) of the data revealed that 23 percent of the total variation in the data was accounted for by the first component, which exceeds the 20 percent criterion established by Reckase (1979) for assuming unidimensionality and that Cognitive Speaking Anxiety (CSA) items were not dimensionally separable from the Psychosomatic Speaking Anxiety (PSA) items.
The study established a pool of 142 items that can be used for a computerized adaptive speaking anxiety severity test. Results of a CAT simulation indicate that a 20–item simulated fixed–length CAT provides better accuracy than that of a P&P test. The computerized adaptive test to assess severity levels of English speaking anxiety proved consistent with the psychometric evidence that suggests that it is feasible to score a single index from the newly constructed anxiety inventory using IRT methods and that this single index score can be compared on common metrics across all items in the item pool. Thus, assessment of speaking anxiety based on CAT may contribute to designing syllabus by establishing an individualized teaching.