title: Question Answering on Patient Medical Records with Private Fine-Tuned
    LLMs
summary: Healthcare systems continuously generate vast amounts of electronic health
  records (EHRs), commonly stored in the Fast Healthcare Interoperability
  Resources (FHIR) standard. Despite the wealth of information in these records,
  their complexity and volume make it difficult for users to retrieve and
  interpret crucial health insights. Recent advances in Large Language Models
  (LLMs) offer a solution, enabling semantic question answering (QA) over medical
  data, allowing users to interact with their health records more effectively.
  However, ensuring privacy and compliance requires edge and private deployments
  of LLMs.
    This paper proposes a novel approach to semantic QA over EHRs by first
  identifying the most relevant FHIR resources for a user query (Task1) and
  subsequently answering the query based on these resources (Task2). We explore
  the performance of privately hosted, fine-tuned LLMs, evaluating them against
  benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that
  fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by
  0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we
  examine advanced aspects of LLM usage, including sequential fine-tuning, model
  self-evaluation (narcissistic evaluation), and the impact of training data size
  on performance. The models and datasets are available here:
  https://huggingface.co/genloop
paper_authors._id: 6797ae961d3cfd7ca5a582a6
paper_authors.name: Sara Kothari
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-27T16:30:51.516Z
paper_authors._id: 66d530c8f040611f7cb342b4
paper_authors.avatarUrl: /avatars/574342d232f897364ab43255e746fc57.svg
paper_authors.fullname: Sara Kothari
paper_authors.type: user
paper_authors.user: sarako
paper_authors._id: 6797ae961d3cfd7ca5a582a7
paper_authors.name: Ayush Gupta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-27T18:59:07.894Z
paper_authors._id: 64b81834f1f8e6ea5841c690
paper_authors.avatarUrl: /avatars/9d8385e687f6bcfe14dff7e6754cc97f.svg
paper_authors.fullname: Ayush Gupta
paper_authors.type: user
paper_authors.user: ayushgs
paper_title: Question Answering on Patient Medical Records with Private Fine-Tuned
    LLMs
paper_summary: Healthcare systems continuously generate vast amounts of electronic health
  records (EHRs), commonly stored in the Fast Healthcare Interoperability
  Resources (FHIR) standard. Despite the wealth of information in these records,
  their complexity and volume make it difficult for users to retrieve and
  interpret crucial health insights. Recent advances in Large Language Models
  (LLMs) offer a solution, enabling semantic question answering (QA) over medical
  data, allowing users to interact with their health records more effectively.
  However, ensuring privacy and compliance requires edge and private deployments
  of LLMs.
    This paper proposes a novel approach to semantic QA over EHRs by first
  identifying the most relevant FHIR resources for a user query (Task1) and
  subsequently answering the query based on these resources (Task2). We explore
  the performance of privately hosted, fine-tuned LLMs, evaluating them against
  benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that
  fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by
  0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we
  examine advanced aspects of LLM usage, including sequential fine-tuning, model
  self-evaluation (narcissistic evaluation), and the impact of training data size
  on performance. The models and datasets are available here:
  https://huggingface.co/genloop
paper_ai_keywords: Large Language Models (LLMs) semantic question answering (QA) FHIR resources privately hosted fine-tuned LLMs GPT-4 sequential fine-tuning model self-evaluation (narcissistic evaluation)