  Computerized Adaptive Testing (CAT) is a widely used, efficient test mode
that adapts to the examinee's proficiency level in the test domain. CAT
requires pre-trained item profiles, for CAT iteratively assesses the student
real-time based on the registered items' profiles, and selects the next item to
administer using candidate items' profiles. However, obtaining such item
profiles is a costly process that involves gathering a large, dense
item-response data, then training a diagnostic model on the collected data. In
this paper, we explore the possibility of leveraging response data collected in
the CAT service. We first show that this poses a unique challenge due to the
inherent selection bias introduced by CAT, i.e., more proficient students will
receive harder questions. Indeed, when naively training the diagnostic model
using CAT response data, we observe that item profiles deviate significantly
from the ground-truth. To tackle the selection bias issue, we propose the
user-wise aggregate influence function method. Our intuition is to filter out
users whose response data is heavily biased in an aggregate manner, as judged
by how much perturbation the added data will introduce during parameter
estimation. This way, we may enhance the performance of CAT while introducing
minimal bias to the item profiles. We provide extensive experiments to
demonstrate the superiority of our proposed method based on the three public
datasets and one dataset that contains real-world CAT response data.

Machine Learning;Computers and Society