  Multiple choice questions (MCQs) are widely used in digital learning systems,
as they allow for automating the assessment process. However, due to the
increased digital literacy of students and the advent of social media
platforms, MCQ tests are widely shared online, and teachers are continuously
challenged to create new questions, which is an expensive and time-consuming
task. A particularly sensitive aspect of MCQ creation is to devise relevant
distractors, i.e., wrong answers that are not easily identifiable as being
wrong. This paper studies how a large existing set of manually created answers
and distractors for questions over a variety of domains, subjects, and
languages can be leveraged to help teachers in creating new MCQs, by the smart
reuse of existing distractors. We built several data-driven models based on
context-aware question and distractor representations, and compared them with
static feature-based models. The proposed models are evaluated with automated
metrics and in a realistic user test with teachers. Both automatic and human
evaluations indicate that context-aware models consistently outperform a static
feature-based approach. For our best-performing context-aware model, on average
3 distractors out of the 10 shown to teachers were rated as high-quality
distractors. We create a performance benchmark, and make it public, to enable
comparison between different approaches and to introduce a more standardized
evaluation of the task. The benchmark contains a test of 298 educational
questions covering multiple subjects & languages and a 77k multilingual pool of
distractor vocabulary for future research.

Computation and Language