DIRAS: Efficient LLM-Assisted Annotation of Document Relevance in Retrieval Augmented Generation
<|reference_start|>DIRAS: Efficient LLM-Assisted Annotation of Document Relevance in Retrieval Augmented Generation: Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information or excessively include irrelevant information? To allay these concerns, it is necessary to annotate domain-specific benchmarks to evaluate information retrieval (IR) performance, as relevance definitions vary across queries and domains. Furthermore, such benchmarks should be cost-efficiently annotated to avoid annotation selection bias. In this paper, we propose DIRAS (Domain-specific Information Retrieval Annotation with Scalability), a manual-annotation-free schema that fine-tunes open-sourced LLMs to annotate relevance labels with calibrated relevance probabilities. Extensive evaluation shows that DIRAS fine-tuned models achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs, and is helpful for real-world RAG development.<|reference_end|>
@article{ni2024diras:,
    title={DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval
  Augmented Generation},
    author={Jingwei Ni, Tobias Schimanski, Meihong Lin, Mrinmaya Sachan, Elliott
  Ash, Markus Leippold},
    journal={arXiv preprint arXiv:2406.14162},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14162},
    primaryClass={cs.IR cs.AI cs.CL}
}