How do you know that? Teaching Generative Language Models to Reference
  Answers to Biomedical Questions
Large language models (LLMs) have recently become the leading source of
answers for users' questions online. Despite their ability to offer eloquent
answers, their accuracy and reliability can pose a significant challenge. This
is especially true for sensitive domains such as biomedicine, where there is a
higher need for factually correct answers. This paper introduces a biomedical
retrieval-augmented generation (RAG) system designed to enhance the reliability
of generated responses. The system is based on a fine-tuned LLM for the
referenced question-answering, where retrieved relevant abstracts from PubMed
are passed to LLM's context as input through a prompt. Its output is an answer
based on PubMed abstracts, where each statement is referenced accordingly,
allowing the users to verify the answer. Our retrieval system achieves an
absolute improvement of 23% compared to the PubMed search engine. Based on the
manual evaluation on a small sample, our fine-tuned LLM component achieves
comparable results to GPT-4 Turbo in referencing relevant abstracts. We make
the dataset used to fine-tune the models and the fine-tuned models based on
Mistral-7B-instruct-v0.1 and v0.2 publicly available.
668d0678fecd7e04836275d2
Bojana Bašaragin
admin_assigned
2024-07-10T10:50:35.775Z
627b83b37a5a8a708ffd18d9
/avatars/6e379274f82f343c94fde99e0a74fd81.svg
Bojana Bašaragin
user
BojanaBas
668d0678fecd7e04836275d3
Adela Ljajić
admin_assigned
2024-07-10T10:50:42.133Z
627da21cba19b19eff696abe
/avatars/870ae4ea50c5c9ef31ec9e772e068995.svg
Adela Ljajić
user
Adelija
668d0678fecd7e04836275d4
Darija Medvecki
admin_assigned
2024-07-10T10:50:47.573Z
64751547b68461d5cf7d5aec
/avatars/909ab45955970f5ff47127262210022e.svg
DarijaM
user
DarijaM
668d0678fecd7e04836275d5
Lorenzo Cassano
668d0678fecd7e04836275d6
Miloš Košprdić
admin_assigned
2024-07-10T10:51:12.705Z
6408811a92033c15073b4d08
/avatars/847c126d0b4fa0834776d73805e19d37.svg
Miloš Košprdić
user
MilosKosRad
668d0678fecd7e04836275d7
Nikola Milošević
claimed_verified
2024-07-10T08:20:00.186Z
6204dcc2522e40b4a18d86eb
https://cdn-avatars.huggingface.co/v1/production/uploads/1644485808126-noauth.jpeg
Nikola
user
nikolamilosevic
How do you know that? Teaching Generative Language Models to Reference
  Answers to Biomedical Questions
Large language models (LLMs) have recently become the leading source of
answers for users' questions online. Despite their ability to offer eloquent
answers, their accuracy and reliability can pose a significant challenge. This
is especially true for sensitive domains such as biomedicine, where there is a
higher need for factually correct answers. This paper introduces a biomedical
retrieval-augmented generation (RAG) system designed to enhance the reliability
of generated responses. The system is based on a fine-tuned LLM for the
referenced question-answering, where retrieved relevant abstracts from PubMed
are passed to LLM's context as input through a prompt. Its output is an answer
based on PubMed abstracts, where each statement is referenced accordingly,
allowing the users to verify the answer. Our retrieval system achieves an
absolute improvement of 23% compared to the PubMed search engine. Based on the
manual evaluation on a small sample, our fine-tuned LLM component achieves
comparable results to GPT-4 Turbo in referencing relevant abstracts. We make
the dataset used to fine-tune the models and the fine-tuned models based on
Mistral-7B-instruct-v0.1 and v0.2 publicly available.
A biomedical retrieval-augmented generation system enhances the reliability of LLM-generated responses by integrating referenced PubMed abstracts, achieving improved accuracy and reference quality compared to existing methods.
LLMs retrieval-augmented generation biomedical domain PubMed fine-tuned LLM GPT-4 Turbo Mistral-7B-instruct-v0.1 Mistral-7B-instruct-v0.2