Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining
  for Clinical LLMs
Large Language Models (LLMs) have demonstrated significant potential in
transforming clinical applications. In this study, we investigate the efficacy
of four techniques in adapting LLMs for clinical use-cases: continuous
pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ
these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale
clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning
dataset of 500 million tokens. Our evaluation across various clinical tasks
reveals the impact of each technique. While continuous pretraining beyond 250
billion tokens yields marginal improvements on its own, it establishes a strong
foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to
enhance generation quality, surprisingly demonstrates additional gains on our
benchmark. Complex prompt engineering methods further enhance performance.
These findings show the importance of tailoring fine-tuning strategies and
exploring innovative techniques to optimize LLM performance in the clinical
domain.
66f261d83a6d90d35e080577
Clément Christophe
claimed_verified
2024-09-24T07:52:16.070Z
628e39f4b1596566033b8d7b
https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg
Clément Christophe
user
cchristophe
66f261d83a6d90d35e080578
Tathagata Raha
66f261d83a6d90d35e080579
Svetlana Maslenkova
claimed_verified
2024-09-25T07:33:57.783Z
6506cfafd55dd4e15caeea09
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uTkC6G8Cj51av4i7kAaI8.png
Svetlana Maslenkova
user
maslenkovas
66f261d83a6d90d35e08057a
Muhammad Umar Salman
66f261d83a6d90d35e08057b
Praveen K Kanithi
claimed_verified
2024-09-24T07:52:13.748Z
65280984b794fe3d06544d77
https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg
Praveenkumar
user
pkanithi
66f261d83a6d90d35e08057c
Marco AF Pimentel
claimed_verified
2024-09-24T14:08:20.395Z
6454faafa13edf669cd74f36
/avatars/355d16e28ca9cf5891368e43bcda6de5.svg
Marco Pimentel
user
mpimentel
66f261d83a6d90d35e08057d
Shadab Khan
Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining
  for Clinical LLMs
Large Language Models (LLMs) have demonstrated significant potential in
transforming clinical applications. In this study, we investigate the efficacy
of four techniques in adapting LLMs for clinical use-cases: continuous
pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ
these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale
clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning
dataset of 500 million tokens. Our evaluation across various clinical tasks
reveals the impact of each technique. While continuous pretraining beyond 250
billion tokens yields marginal improvements on its own, it establishes a strong
foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to
enhance generation quality, surprisingly demonstrates additional gains on our
benchmark. Complex prompt engineering methods further enhance performance.
These findings show the importance of tailoring fine-tuning strategies and
exploring innovative techniques to optimize LLM performance in the clinical
domain.
Four techniques—continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering—improve Large Language Models for clinical applications, with NEFTune and advanced prompt engineering yielding significant performance enhancements.
continuous pretraining instruct fine-tuning NEFTune prompt engineering