Dr. LLaMA: Improving Small Language Models in Domain-Specific QA via
  Generative Data Augmentation
Large Language Models (LLMs) have made significant strides in natural
language processing but face challenges in terms of computational expense and
inefficiency as they grow in size, especially in domain-specific tasks. Small
Language Models (SLMs), on the other hand, often struggle in these tasks due to
limited capacity and training data. In this paper, we introduce Dr. LLaMA, a
method for improving SLMs through generative data augmentation using LLMs,
focusing on medical question-answering tasks and the PubMedQA dataset. Our
findings indicate that LLMs effectively refine and diversify existing
question-answer pairs, resulting in improved performance of a much smaller
model on domain-specific QA datasets after fine-tuning. This study highlights
the challenges of using LLMs for domain-specific question answering and
suggests potential research directions to address these limitations, ultimately
aiming to create more efficient and capable models for specialized
applications. We have also made our code available for interested researchers
6462e7d1d1ccd517f46a4789
Zhen Guo
claimed_verified
2023-05-16T12:25:16.528Z
638e4e66629b4d0a62ce1bf3
https://cdn-avatars.huggingface.co/v1/production/uploads/638e4e66629b4d0a62ce1bf3/s7uQ2qmee2CaXfjZDOovZ.jpeg
Zhen Guo
user
zguo0525
6462e7d1d1ccd517f46a478a
Peiqi Wang
6462e7d1d1ccd517f46a478b
Yanwei Wang
claimed_verified
2023-08-01T18:23:38.414Z
61020fd087b0cc005170cf32
/avatars/22128384b38972ba73b77f57c52a2a13.svg
Felix Wang
user
felixw
6462e7d1d1ccd517f46a478c
Shangdi Yu
claimed_verified
2023-05-16T15:58:20.149Z
6463a6cb4bf912292226962e
/avatars/df70538d579778c531b4665881adb623.svg
Shangdi Yu
user
yushangdi
Dr. LLaMA: Improving Small Language Models in Domain-Specific QA via
  Generative Data Augmentation
Large Language Models (LLMs) have made significant strides in natural
language processing but face challenges in terms of computational expense and
inefficiency as they grow in size, especially in domain-specific tasks. Small
Language Models (SLMs), on the other hand, often struggle in these tasks due to
limited capacity and training data. In this paper, we introduce Dr. LLaMA, a
method for improving SLMs through generative data augmentation using LLMs,
focusing on medical question-answering tasks and the PubMedQA dataset. Our
findings indicate that LLMs effectively refine and diversify existing
question-answer pairs, resulting in improved performance of a much smaller
model on domain-specific QA datasets after fine-tuning. This study highlights
the challenges of using LLMs for domain-specific question answering and
suggests potential research directions to address these limitations, ultimately
aiming to create more efficient and capable models for specialized
applications. We have also made our code available for interested researchers
Dr. LLaMA uses generative data augmentation from Large Language Models to improve Small Language Models in domain-specific question-answering tasks, specifically enhancing performance on medical datasets like PubMedQA.
Large Language Models Small Language Models generative data augmentation medical question-answering PubMedQA fine-tuning