Data Checklist: On Unit-Testing Datasets with Usable Information
<|reference_start|>Data Checklist: On Unit-Testing Datasets with Usable Information: Model checklists (Ribeiro et al., 2020) have emerged as a useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering. However, despite datasets being a key determinant of model behavior, evaluating datasets, e.g., for the existence of annotation artifacts, is largely done ad hoc, once a problem in model behavior has already been found downstream. In this work, we take a more principled approach to unit-testing datasets by proposing a taxonomy based on the V-information literature. We call a collection of such unit tests a data checklist. Using a checklist, not only are we able to recover known artifacts in well-known datasets such as SNLI, but we also discover previously unknown artifacts in preference datasets for LLM alignment. Data checklists further enable a new kind of data filtering, which we use to improve the efficacy and data efficiency of preference alignment.<|reference_end|>
@article{zhang2024data,
    title={Data Checklist: On Unit-Testing Datasets with Usable Information},
    author={Heidi C. Zhang, Shabnam Behzad, Kawin Ethayarajh, Dan Jurafsky},
    journal={arXiv preprint arXiv:2408.02919},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.02919},
    primaryClass={cs.CL}
}