NLLB-E5: A Scalable Multilingual Retrieval Model
<|reference_start|>NLLB-E5: A Scalable Multilingual Retrieval Model: Despite significant progress in multilingual information retrieval, the lack of models capable of effectively supporting multiple languages, particularly low-resource like Indic languages, remains a critical challenge. This paper presents NLLB-E5: A Scalable Multilingual Retrieval Model. NLLB-E5 leverages the in-built multilingual capabilities in the NLLB encoder for translation tasks. It proposes a distillation approach from multilingual retriever E5 to provide a zero-shot retrieval approach handling multiple languages, including all major Indic languages, without requiring multilingual training data. We evaluate the model on a comprehensive suite of existing benchmarks, including Hindi-BEIR, highlighting its robust performance across diverse languages and tasks. Our findings uncover task and domain-specific challenges, providing valuable insights into the retrieval performance, especially for low-resource languages. NLLB-E5 addresses the urgent need for an inclusive, scalable, and language-agnostic text retrieval model, advancing the field of multilingual information access and promoting digital inclusivity for millions of users globally.<|reference_end|>
@article{acharya2024benchmarking,
    title={Benchmarking and Building Zero-Shot Hindi Retrieval Model with
  Hindi-BEIR and NLLB-E5},
    author={Arkadeep Acharya, Rudra Murthy, Vishwajeet Kumar, Jaydeep Sen},
    journal={arXiv preprint arXiv:2409.05401},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05401},
    primaryClass={cs.IR cs.CL}
}