Question Answering on Patient Medical Records with Private Fine-Tuned
  LLMs
Healthcare systems continuously generate vast amounts of electronic health
records (EHRs), commonly stored in the Fast Healthcare Interoperability
Resources (FHIR) standard. Despite the wealth of information in these records,
their complexity and volume make it difficult for users to retrieve and
interpret crucial health insights. Recent advances in Large Language Models
(LLMs) offer a solution, enabling semantic question answering (QA) over medical
data, allowing users to interact with their health records more effectively.
However, ensuring privacy and compliance requires edge and private deployments
of LLMs.
  This paper proposes a novel approach to semantic QA over EHRs by first
identifying the most relevant FHIR resources for a user query (Task1) and
subsequently answering the query based on these resources (Task2). We explore
the performance of privately hosted, fine-tuned LLMs, evaluating them against
benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that
fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by
0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we
examine advanced aspects of LLM usage, including sequential fine-tuning, model
self-evaluation (narcissistic evaluation), and the impact of training data size
on performance. The models and datasets are available here:
https://huggingface.co/genloop
6797ae961d3cfd7ca5a582a6
Sara Kothari
extracted_confirmed
2025-01-27T16:30:51.516Z
66d530c8f040611f7cb342b4
/avatars/574342d232f897364ab43255e746fc57.svg
Sara Kothari
user
sarako
6797ae961d3cfd7ca5a582a7
Ayush Gupta
extracted_confirmed
2025-01-27T18:59:07.894Z
64b81834f1f8e6ea5841c690
/avatars/9d8385e687f6bcfe14dff7e6754cc97f.svg
Ayush Gupta
user
ayushgs
Question Answering on Patient Medical Records with Private Fine-Tuned
  LLMs
Healthcare systems continuously generate vast amounts of electronic health
records (EHRs), commonly stored in the Fast Healthcare Interoperability
Resources (FHIR) standard. Despite the wealth of information in these records,
their complexity and volume make it difficult for users to retrieve and
interpret crucial health insights. Recent advances in Large Language Models
(LLMs) offer a solution, enabling semantic question answering (QA) over medical
data, allowing users to interact with their health records more effectively.
However, ensuring privacy and compliance requires edge and private deployments
of LLMs.
  This paper proposes a novel approach to semantic QA over EHRs by first
identifying the most relevant FHIR resources for a user query (Task1) and
subsequently answering the query based on these resources (Task2). We explore
the performance of privately hosted, fine-tuned LLMs, evaluating them against
benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that
fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by
0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we
examine advanced aspects of LLM usage, including sequential fine-tuning, model
self-evaluation (narcissistic evaluation), and the impact of training data size
on performance. The models and datasets are available here:
https://huggingface.co/genloop
A fine-tuned LLM approach for semantic QA over EHRs using FHIR resources outperforms GPT-4 in both task performance and model size, with evaluation techniques including sequential fine-tuning and self-evaluation.
Large Language Models semantic question answering EHRs FHIR privately hosted fine-tuned LLMs GPT-4 GPT-4o sequential fine-tuning model self-evaluation F1 score Meteor Task