Abstract

Designing a high-quality multiple-choice test is a challenging task. Typically, to validate a test, this must be administered to a sample of the target population, allowing one to estimate the difficulty of each question and its consistency. In several scenarios, this administration is costly and time-consuming, so predicting the difficulty of multiple-choice questions before field testing could reduce costs and time during the test validation process. In this article, we propose three deep-learning approaches which aim to reduce the resources required to estimate the difficulty of multiple-choice questions during test development of high-stakes tests. These data-driven approaches use Neural Network architectures such as Recurrent Neural Networks (RNN), Bidirectional Long Short-term Memory (BiLSTM), and Bidirectional Encoder Representations for Transformers (BERT). The models are trained on a data source built with a sample of the standardized high-stakes exams for university admissions in Chile. Our approaches consider different configurations specific to each architecture and a set of features that represent the readability level and the similarities between the response options. The results show that BiLSTM performs best and is the most suitable model for the task, even though it could be considered outdated by the appearance of contemporary architectures. Finally, we elaborate on how this data-driven approach might be improved.