Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases
<|reference_start|>Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases: We proposed an end-to-end system design towards utilizing Retrieval Augmented Generation (RAG) to improve the factual accuracy of Large Language Models (LLMs) for domain-specific and time-sensitive queries related to private knowledge-bases. Our system integrates RAG pipeline with upstream datasets processing and downstream performance evaluation. Addressing the challenge of LLM hallucinations, we finetune models with a curated dataset which originates from CMU's extensive resources and annotated with the teacher model. Our experiments demonstrate the system's effectiveness in generating more accurate answers to domain-specific and time-sensitive inquiries. The results also revealed the limitations of fine-tuning LLMs with small-scale and skewed datasets. This research highlights the potential of RAG systems in augmenting LLMs with external datasets for improved performance in knowledge-intensive tasks. Our code and models are available on Github.<|reference_end|>
@article{li2024enhancing,
    title={Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A
  Case Study on Domain-Specific Queries in Private Knowledge-Bases},
    author={Jiarui Li and Ye Yuan and Zehua Zhang},
    journal={arXiv preprint arXiv:2403.10446},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10446},
    primaryClass={cs.CL cs.LG}
}