Distilling Large Language Models for Biomedical Knowledge Extraction: A
  Case Study on Adverse Drug Events
Large language models (LLMs), such as GPT-4, have demonstrated remarkable
capabilities across a wide range of tasks, including health applications. In
this paper, we study how LLMs can be used to scale biomedical knowledge
curation. We find that while LLMs already possess decent competency in
structuring biomedical text, by distillation into a task-specific student model
through self-supervised learning, substantial gains can be attained over
out-of-box LLMs, with additional advantages such as cost, efficiency, and
white-box model access.
  We conduct a case study on adverse drug event (ADE) extraction, which is an
important area for improving care. On standard ADE extraction evaluation, a
GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised
state-of-the-art models without using any labeled data. Despite being over
1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by
over 6 absolute points in F1 and GPT-4 by over 5 absolute points.
  Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)
and ADE extraction architecture shed light on best practice for biomedical
knowledge extraction. Similar gains were attained by distillation for other
standard biomedical knowledge extraction tasks such as gene-disease
associations and protected health information, further illustrating the promise
of this approach.
64b09a2d4db8a01408c2f5f2
Yu Gu
claimed_verified
2023-08-02T20:30:35.329Z
645a02972abf6165a3ba5df8
/avatars/6166e4f213b9020ba99f853cb04f8db0.svg
Yu Gu
user
aidenygu
64b09a2d4db8a01408c2f5f3
Sheng Zhang
claimed_verified
2024-06-17T07:35:01.740Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
64b09a2d4db8a01408c2f5f4
Naoto Usuyama
claimed_verified
2023-07-14T19:33:10.356Z
5f243561e923d665e616263f
https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
Naoto Usuyama
user
naotous
64b09a2d4db8a01408c2f5f5
Yonas Woldesenbet
64b09a2d4db8a01408c2f5f6
Cliff Wong
claimed_verified
2023-07-18T11:11:13.922Z
64b668e8575d40032dd3005e
/avatars/c4eeef2f911589fa93a427353da7cec7.svg
Cliff Wong
user
cliffwong
64b09a2d4db8a01408c2f5f7
Praneeth Sanapathi
64b09a2d4db8a01408c2f5f8
Mu Wei
64b09a2d4db8a01408c2f5f9
Naveen Valluri
admin_assigned
2023-07-17T10:33:23.933Z
60538cee34c3e14ef093c094
/avatars/b80880f342194920b31be69a024fb59b.svg
Naveen Valluri
user
nvalluri
64b09a2d4db8a01408c2f5fa
Erika Strandberg
64b09a2d4db8a01408c2f5fb
Tristan Naumann
admin_assigned
2023-07-17T10:34:01.481Z
5e5870466bc35159a08ca572
https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
Tristan Naumann
user
tnaumann
64b09a2d4db8a01408c2f5fc
Hoifung Poon
Distilling Large Language Models for Biomedical Knowledge Extraction: A
  Case Study on Adverse Drug Events
Large language models (LLMs), such as GPT-4, have demonstrated remarkable
capabilities across a wide range of tasks, including health applications. In
this paper, we study how LLMs can be used to scale biomedical knowledge
curation. We find that while LLMs already possess decent competency in
structuring biomedical text, by distillation into a task-specific student model
through self-supervised learning, substantial gains can be attained over
out-of-box LLMs, with additional advantages such as cost, efficiency, and
white-box model access.
  We conduct a case study on adverse drug event (ADE) extraction, which is an
important area for improving care. On standard ADE extraction evaluation, a
GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised
state-of-the-art models without using any labeled data. Despite being over
1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by
over 6 absolute points in F1 and GPT-4 by over 5 absolute points.
  Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)
and ADE extraction architecture shed light on best practice for biomedical
knowledge extraction. Similar gains were attained by distillation for other
standard biomedical knowledge extraction tasks such as gene-disease
associations and protected health information, further illustrating the promise
of this approach.
Distillation of large language models into task-specific student models via self-supervised learning enhances biomedical knowledge extraction tasks with cost and efficiency benefits.
large language models LLMs GPT-4 biomedical knowledge curation distillation self-supervised learning adverse drug event extraction PubMedBERT BioGPT gene-disease associations protected health information F1 score