title: Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining
    for Clinical LLMs
summary: Large Language Models (LLMs) have demonstrated significant potential in
  transforming clinical applications. In this study, we investigate the efficacy
  of four techniques in adapting LLMs for clinical use-cases: continuous
  pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ
  these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale
  clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning
  dataset of 500 million tokens. Our evaluation across various clinical tasks
  reveals the impact of each technique. While continuous pretraining beyond 250
  billion tokens yields marginal improvements on its own, it establishes a strong
  foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to
  enhance generation quality, surprisingly demonstrates additional gains on our
  benchmark. Complex prompt engineering methods further enhance performance.
  These findings show the importance of tailoring fine-tuning strategies and
  exploring innovative techniques to optimize LLM performance in the clinical
  domain.
paper_authors._id: 66f261d83a6d90d35e080577
paper_authors.name: Clément Christophe
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:16.070Z
paper_authors._id: 628e39f4b1596566033b8d7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg
paper_authors.fullname: Clément Christophe
paper_authors.type: user
paper_authors.user: cchristophe
paper_authors._id: 66f261d83a6d90d35e080578
paper_authors.name: Tathagata Raha
paper_authors._id: 66f261d83a6d90d35e080579
paper_authors.name: Svetlana Maslenkova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:33:57.783Z
paper_authors._id: 6506cfafd55dd4e15caeea09
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uTkC6G8Cj51av4i7kAaI8.png
paper_authors.fullname: Svetlana Maslenkova
paper_authors.type: user
paper_authors.user: maslenkovas
paper_authors._id: 66f261d83a6d90d35e08057a
paper_authors.name: Muhammad Umar Salman
paper_authors._id: 66f261d83a6d90d35e08057b
paper_authors.name: Praveen K Kanithi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:13.748Z
paper_authors._id: 65280984b794fe3d06544d77
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg
paper_authors.fullname: Praveenkumar
paper_authors.type: user
paper_authors.user: pkanithi
paper_authors._id: 66f261d83a6d90d35e08057c
paper_authors.name: Marco AF Pimentel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T14:08:20.395Z
paper_authors._id: 6454faafa13edf669cd74f36
paper_authors.avatarUrl: /avatars/355d16e28ca9cf5891368e43bcda6de5.svg
paper_authors.fullname: Marco Pimentel
paper_authors.type: user
paper_authors.user: mpimentel
paper_authors._id: 66f261d83a6d90d35e08057d
paper_authors.name: Shadab Khan
paper_title: Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining
    for Clinical LLMs
paper_summary: Large Language Models (LLMs) have demonstrated significant potential in
  transforming clinical applications. In this study, we investigate the efficacy
  of four techniques in adapting LLMs for clinical use-cases: continuous
  pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ
  these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale
  clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning
  dataset of 500 million tokens. Our evaluation across various clinical tasks
  reveals the impact of each technique. While continuous pretraining beyond 250
  billion tokens yields marginal improvements on its own, it establishes a strong
  foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to
  enhance generation quality, surprisingly demonstrates additional gains on our
  benchmark. Complex prompt engineering methods further enhance performance.
  These findings show the importance of tailoring fine-tuning strategies and
  exploring innovative techniques to optimize LLM performance in the clinical
  domain.
paper_ai_summary: Four techniques—continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering—improve Large Language Models for clinical applications, with NEFTune and advanced prompt engineering yielding significant performance enhancements.
paper_ai_keywords: continuous pretraining instruct fine-tuning NEFTune prompt engineering