FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation
<|reference_start|>FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation: Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks -- primarily sentence- and discourse-level translation -- between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.<|reference_end|>
@article{tanzer2024fleurs-asl:,
    title={FLEURS-ASL: Including American Sign Language in Massively Multilingual
  Multitask Evaluation},
    author={Garrett Tanzer},
    journal={arXiv preprint arXiv:2408.13585},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.13585},
    primaryClass={cs.CL cs.CV}
}