title: Dr. LLaMA: Improving Small Language Models in Domain-Specific QA via
    Generative Data Augmentation
summary: Large Language Models (LLMs) have made significant strides in natural
  language processing but face challenges in terms of computational expense and
  inefficiency as they grow in size, especially in domain-specific tasks. Small
  Language Models (SLMs), on the other hand, often struggle in these tasks due to
  limited capacity and training data. In this paper, we introduce Dr. LLaMA, a
  method for improving SLMs through generative data augmentation using LLMs,
  focusing on medical question-answering tasks and the PubMedQA dataset. Our
  findings indicate that LLMs effectively refine and diversify existing
  question-answer pairs, resulting in improved performance of a much smaller
  model on domain-specific QA datasets after fine-tuning. This study highlights
  the challenges of using LLMs for domain-specific question answering and
  suggests potential research directions to address these limitations, ultimately
  aiming to create more efficient and capable models for specialized
  applications. We have also made our code available for interested researchers
paper_authors._id: 6462e7d1d1ccd517f46a4789
paper_authors.name: Zhen Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T12:25:16.528Z
paper_authors._id: 638e4e66629b4d0a62ce1bf3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638e4e66629b4d0a62ce1bf3/s7uQ2qmee2CaXfjZDOovZ.jpeg
paper_authors.fullname: Zhen Guo
paper_authors.type: user
paper_authors.user: zguo0525
paper_authors._id: 6462e7d1d1ccd517f46a478a
paper_authors.name: Peiqi Wang
paper_authors._id: 6462e7d1d1ccd517f46a478b
paper_authors.name: Yanwei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T18:23:38.414Z
paper_authors._id: 61020fd087b0cc005170cf32
paper_authors.avatarUrl: /avatars/5e8038e9ef372d81f57bb87d7636476b.svg
paper_authors.fullname: Felix Wang
paper_authors.type: user
paper_authors.user: felixw
paper_authors._id: 6462e7d1d1ccd517f46a478c
paper_authors.name: Shangdi Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:58:20.149Z
paper_authors._id: 6463a6cb4bf912292226962e
paper_authors.avatarUrl: /avatars/df70538d579778c531b4665881adb623.svg
paper_authors.fullname: Shangdi Yu
paper_authors.type: user
paper_authors.user: yushangdi
paper_title: Dr. LLaMA: Improving Small Language Models in Domain-Specific QA via
    Generative Data Augmentation
paper_summary: Large Language Models (LLMs) have made significant strides in natural
  language processing but face challenges in terms of computational expense and
  inefficiency as they grow in size, especially in domain-specific tasks. Small
  Language Models (SLMs), on the other hand, often struggle in these tasks due to
  limited capacity and training data. In this paper, we introduce Dr. LLaMA, a
  method for improving SLMs through generative data augmentation using LLMs,
  focusing on medical question-answering tasks and the PubMedQA dataset. Our
  findings indicate that LLMs effectively refine and diversify existing
  question-answer pairs, resulting in improved performance of a much smaller
  model on domain-specific QA datasets after fine-tuning. This study highlights
  the challenges of using LLMs for domain-specific question answering and
  suggests potential research directions to address these limitations, ultimately
  aiming to create more efficient and capable models for specialized
  applications. We have also made our code available for interested researchers
paper_ai_keywords: generative data augmentation fine-tuning medical question-answering PubMedQA dataset domain-specific QA datasets