DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
<|reference_start|>DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages: Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is language variation in the form of non-standard dialects or language varieties (hereafter, varieties). Most NLP benchmarks are limited to standard language varieties. To fill this gap, we propose DIALECTBENCH, the first-ever large-scale benchmark for NLP on varieties, which aggregates an extensive set of task-varied variety datasets (10 text-level tasks covering 281 varieties). This allows for a comprehensive evaluation of NLP system performance on different language varieties. We provide substantial evidence of performance disparities between standard and non-standard language varieties, and we also identify language clusters with large performance divergence across tasks. We believe DIALECTBENCH provides a comprehensive view of the current state of NLP for language varieties and one step towards advancing it further. Code/data: https://github.com/ffaisal93/DialectBench<|reference_end|>
@article{faisal2024dialectbench:,
    title={DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and
  Closely-Related Languages},
    author={Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David
  Chiang, Yulia Tsvetkov, Antonios Anastasopoulos},
    journal={arXiv preprint arXiv:2403.11009},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11009},
    primaryClass={cs.CL cs.AI}
}