Multilingual Evaluation of Long Context Retrieval and Reasoning
<|reference_start|>Multilingual Evaluation of Long Context Retrieval and Reasoning: Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We comprehensively evaluate several long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.<|reference_end|>
@article{agrawal2024evaluating,
    title={Evaluating Multilingual Long-Context Models for Retrieval and Reasoning},
    author={Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel,
  Russell Scheinberg},
    journal={arXiv preprint arXiv:2409.18006},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.18006},
    primaryClass={cs.CL}
}