title: Training Task Experts through Retrieval Based Distillation
summary: One of the most reliable ways to create deployable models for specialized
  tasks is to obtain an adequate amount of high-quality task-specific data.
  However, for specialized tasks, often such datasets do not exist. Existing
  methods address this by creating such data from large language models (LLMs)
  and then distilling such knowledge into smaller models. However, these methods
  are limited by the quality of the LLMs output, and tend to generate repetitive
  or incorrect data. In this work, we present Retrieval Based Distillation
  (ReBase), a method that first retrieves data from rich online sources and then
  transforms them into domain-specific data. This method greatly enhances data
  diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills
  the reasoning capacity of LLMs. We test our method on 4 benchmarks and results
  show that our method significantly improves performance by up to 7.8% on SQuAD,
  1.37% on MNLI, and 1.94% on BigBench-Hard.
paper_authors._id: 668d62cdc04604b679487253
paper_authors.name: Jiaxin Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:36:12.396Z
paper_authors._id: 64bdc1e8d05a97d722b2c1f6
paper_authors.avatarUrl: /avatars/b04cbe17e7cd0a1f5bf0e8e008640a1d.svg
paper_authors.fullname: Jiaxin Ge
paper_authors.type: user
paper_authors.user: para-lost
paper_authors._id: 668d62cdc04604b679487254
paper_authors.name: Xueying Jia
paper_authors._id: 668d62cdc04604b679487255
paper_authors.name: Vijay Viswanathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:37:02.268Z
paper_authors._id: 603c44e677a2a899efe25828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1614562504843-noauth.jpeg
paper_authors.fullname: Vijay Viswanathan
paper_authors.type: user
paper_authors.user: viswavi
paper_authors._id: 668d62cdc04604b679487256
paper_authors.name: Hongyin Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:05.252Z
paper_authors._id: 62bc495016897fa76cdcda18
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
paper_authors.fullname: Hongyin Luo
paper_authors.type: user
paper_authors.user: luohy
paper_authors._id: 668d62cdc04604b679487257
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:37:09.136Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: Training Task Experts through Retrieval Based Distillation
paper_summary: One of the most reliable ways to create deployable models for specialized
  tasks is to obtain an adequate amount of high-quality task-specific data.
  However, for specialized tasks, often such datasets do not exist. Existing
  methods address this by creating such data from large language models (LLMs)
  and then distilling such knowledge into smaller models. However, these methods
  are limited by the quality of the LLMs output, and tend to generate repetitive
  or incorrect data. In this work, we present Retrieval Based Distillation
  (ReBase), a method that first retrieves data from rich online sources and then
  transforms them into domain-specific data. This method greatly enhances data
  diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills
  the reasoning capacity of LLMs. We test our method on 4 benchmarks and results
  show that our method significantly improves performance by up to 7.8% on SQuAD,
  1.37% on MNLI, and 1.94% on BigBench-Hard.
paper_ai_keywords: Retrieval Based Distillation ReBase Chain-of-Thought reasoning pre-trained large language models encoder data distillation SQuAD MNLI BigBench-Hard adversarial performance improvement