title: Distilling Large Language Models for Biomedical Knowledge Extraction: A
    Case Study on Adverse Drug Events
summary: Large language models (LLMs), such as GPT-4, have demonstrated remarkable
  capabilities across a wide range of tasks, including health applications. In
  this paper, we study how LLMs can be used to scale biomedical knowledge
  curation. We find that while LLMs already possess decent competency in
  structuring biomedical text, by distillation into a task-specific student model
  through self-supervised learning, substantial gains can be attained over
  out-of-box LLMs, with additional advantages such as cost, efficiency, and
  white-box model access.
    We conduct a case study on adverse drug event (ADE) extraction, which is an
  important area for improving care. On standard ADE extraction evaluation, a
  GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised
  state-of-the-art models without using any labeled data. Despite being over
  1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by
  over 6 absolute points in F1 and GPT-4 by over 5 absolute points.
    Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)
  and ADE extraction architecture shed light on best practice for biomedical
  knowledge extraction. Similar gains were attained by distillation for other
  standard biomedical knowledge extraction tasks such as gene-disease
  associations and protected health information, further illustrating the promise
  of this approach.
paper_authors._id: 64b09a2d4db8a01408c2f5f2
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-02T20:30:35.329Z
paper_authors._id: 645a02972abf6165a3ba5df8
paper_authors.avatarUrl: /avatars/6166e4f213b9020ba99f853cb04f8db0.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: aidenygu
paper_authors._id: 64b09a2d4db8a01408c2f5f3
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:35:01.740Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 64b09a2d4db8a01408c2f5f4
paper_authors.name: Naoto Usuyama
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-14T19:33:10.356Z
paper_authors._id: 5f243561e923d665e616263f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
paper_authors.fullname: Naoto Usuyama
paper_authors.type: user
paper_authors.user: naotous
paper_authors._id: 64b09a2d4db8a01408c2f5f5
paper_authors.name: Yonas Woldesenbet
paper_authors._id: 64b09a2d4db8a01408c2f5f6
paper_authors.name: Cliff Wong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T11:11:13.922Z
paper_authors._id: 64b668e8575d40032dd3005e
paper_authors.avatarUrl: /avatars/c4eeef2f911589fa93a427353da7cec7.svg
paper_authors.fullname: Cliff Wong
paper_authors.type: user
paper_authors.user: cliffwong
paper_authors._id: 64b09a2d4db8a01408c2f5f7
paper_authors.name: Praneeth Sanapathi
paper_authors._id: 64b09a2d4db8a01408c2f5f8
paper_authors.name: Mu Wei
paper_authors._id: 64b09a2d4db8a01408c2f5f9
paper_authors.name: Naveen Valluri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:33:23.933Z
paper_authors._id: 60538cee34c3e14ef093c094
paper_authors.avatarUrl: /avatars/b80880f342194920b31be69a024fb59b.svg
paper_authors.fullname: Naveen Valluri
paper_authors.type: user
paper_authors.user: nvalluri
paper_authors._id: 64b09a2d4db8a01408c2f5fa
paper_authors.name: Erika Strandberg
paper_authors._id: 64b09a2d4db8a01408c2f5fb
paper_authors.name: Tristan Naumann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:34:01.481Z
paper_authors._id: 5e5870466bc35159a08ca572
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e5870466bc35159a08ca572/pT6gEEs8RLRJGeM-faNWj.jpeg
paper_authors.fullname: Tristan Naumann
paper_authors.type: user
paper_authors.user: tnaumann
paper_authors._id: 64b09a2d4db8a01408c2f5fc
paper_authors.name: Hoifung Poon
paper_title: Distilling Large Language Models for Biomedical Knowledge Extraction: A
    Case Study on Adverse Drug Events
paper_summary: Large language models (LLMs), such as GPT-4, have demonstrated remarkable
  capabilities across a wide range of tasks, including health applications. In
  this paper, we study how LLMs can be used to scale biomedical knowledge
  curation. We find that while LLMs already possess decent competency in
  structuring biomedical text, by distillation into a task-specific student model
  through self-supervised learning, substantial gains can be attained over
  out-of-box LLMs, with additional advantages such as cost, efficiency, and
  white-box model access.
    We conduct a case study on adverse drug event (ADE) extraction, which is an
  important area for improving care. On standard ADE extraction evaluation, a
  GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised
  state-of-the-art models without using any labeled data. Despite being over
  1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by
  over 6 absolute points in F1 and GPT-4 by over 5 absolute points.
    Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)
  and ADE extraction architecture shed light on best practice for biomedical
  knowledge extraction. Similar gains were attained by distillation for other
  standard biomedical knowledge extraction tasks such as gene-disease
  associations and protected health information, further illustrating the promise
  of this approach.
paper_ai_summary: Distillation of large language models into task-specific student models via self-supervised learning enhances biomedical knowledge extraction tasks with cost and efficiency benefits.
paper_ai_keywords: large language models LLMs GPT-4 biomedical knowledge curation distillation self-supervised learning adverse drug event extraction PubMedBERT BioGPT gene-disease associations protected health information F1 score