Reassessing assessment : using Bayesian methods to examine the efficacy of multiple-select multiple-choice items

Standardized tests play a central role in nearly all modern schools in the United States. Recent research has indicated that many of these standardized tests do not effectively measure students' underlying knowledge of a subject. Additional research must be completed to find innovative and effective items that have not only successfully measure student ability, but also provide educators with information about those abilities so they can effectively address student deficiencies. 371 participants were recruited from introductory statistics courses and given an assessment that contained both traditional single-response multiple choice items and multiple-select multiple-choice items. The participants were also randomly divided into two conditions, one where the directions told them they should select single responses for the single-response items and multiple responses for the multi-select items. The second condition instructed participants that all items could have multiple responses. All items were analyzed using traditional IRT and Bayesian IRT methods. Items analyzed with Bayesian methods have yielded significant statistical differences in item difficulty parameters versus traditional 1-PL Item Response Theory models, especially for items that had high difficulty parameter values. Further validation of the participants results was done with Bayesian regression models, which indicated that there was a moderately weak relationship between student performance on standard in-class examinations versus results on the instrument used in this study, which indicates that participant performance is not strongly related to performance on in-class exams. This indicates that further analysis of multiple-select items is warranted. Additionally, since multiple-select items are multifaceted, they have the potential to provide instructors with the ability to see if a student has full, partial, or no understanding of a concept as opposed to traditional single response multiple-choice items