Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi
<|reference_start|>Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi: Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, there is a lack of comprehensive benchmark for evaluating retrieval models in Hindi. To address this gap, we introduce the Hindi version of the BEIR benchmark, which includes a subset of English BEIR datasets translated to Hindi, existing Hindi retrieval datasets, and synthetically created datasets for retrieval. The benchmark is comprised of $15$ datasets spanning across $8$ distinct tasks. We evaluate state-of-the-art multilingual retrieval models on this benchmark to identify task and domain-specific challenges and their impact on retrieval performance. By releasing this benchmark and a set of relevant baselines, we enable researchers to understand the limitations and capabilities of current Hindi retrieval models, promoting advancements in this critical area. The datasets from Hindi-BEIR are publicly available.<|reference_end|>
@article{acharya2024hindi-beir,
    title={Hindi-BEIR : A Large Scale Retrieval Benchmark in Hindi},
    author={Arkadeep Acharya, Rudra Murthy, Vishwajeet Kumar, Jaydeep Sen},
    journal={arXiv preprint arXiv:2408.09437},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09437},
    primaryClass={cs.IR cs.CL}
}