Intrinsic Evaluation of RAG Systems for Deep-Logic Questions
<|reference_start|>Intrinsic Evaluation of RAG Systems for Deep-Logic Questions: We introduce the Overall Performance Index (OPI), an intrinsic metric to evaluate retrieval-augmented generation (RAG) mechanisms for applications involving deep-logic queries. OPI is computed as the harmonic mean of two key metrics: the Logical-Relation Correctness Ratio and the average of BERT embedding similarity scores between ground-truth and generated answers. We apply OPI to assess the performance of LangChain, a popular RAG tool, using a logical relations classifier fine-tuned from GPT-4o on the RAG-Dataset-12000 from Hugging Face. Our findings show a strong correlation between BERT embedding similarity scores and extrinsic evaluation scores. Among the commonly used retrievers, the cosine similarity retriever using BERT-based embeddings outperforms others, while the Euclidean distance-based retriever exhibits the weakest performance. Furthermore, we demonstrate that combining multiple retrievers, either algorithmically or by merging retrieved sentences, yields superior performance compared to using any single retriever alone.<|reference_end|>
@article{hu2024intrinsic,
    title={Intrinsic Evaluation of RAG Systems for Deep-Logic Questions},
    author={Junyi Hu, You Zhou, Jie Wang},
    journal={arXiv preprint arXiv:2410.02932},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02932},
    primaryClass={cs.AI}
}