title: Ensemble-Instruct: Generating Instruction-Tuning Data with a
    Heterogeneous Mixture of LMs
summary: Using in-context learning (ICL) for data generation, techniques such as
  Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023)
  can train strong conversational agents with only a small amount of human
  supervision. One limitation of these approaches is that they resort to very
  large language models (around 175B parameters) that are also proprietary and
  non-public. Here we explore the application of such techniques to language
  models that are much smaller (around 10B--40B parameters) and have permissive
  licenses. We find the Self-Instruct approach to be less effective at these
  sizes and propose new ICL methods that draw on two main ideas: (a)
  Categorization and simplification of the ICL templates to make prompt learning
  easier for the LM, and (b) Ensembling over multiple LM outputs to help select
  high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct
  seed tasks and employs separate pipelines for instructions that require an
  input and instructions that do not. Empirical investigations with different LMs
  show that: (1) Our proposed method yields higher-quality instruction tuning
  data than Self-Instruct, (2) It improves performances of both vanilla and
  instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned
  LMs generate more useful outputs than their larger un-tuned counterparts. Our
  codebase is available at https://github.com/IBM/ensemble-instruct.
paper_authors._id: 65374541fa9f02750d26d086
paper_authors.name: Young-Suk Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:51:10.070Z
paper_authors._id: 64ba9290cefbfd87d391b11d
paper_authors.avatarUrl: /avatars/557e7b850c7cfacd81738bb33d7e9b16.svg
paper_authors.fullname: Young-Suk Lee
paper_authors.type: user
paper_authors.user: samiangel
paper_authors._id: 65374541fa9f02750d26d087
paper_authors.name: Md Arafat Sultan
paper_authors._id: 65374541fa9f02750d26d088
paper_authors.name: Yousef El-Kurdi
paper_authors._id: 65374541fa9f02750d26d089
paper_authors.name: Tahira Naseem Asim Munawar
paper_authors._id: 65374541fa9f02750d26d08a
paper_authors.name: Radu Florian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:51:32.325Z
paper_authors._id: 642ae74b25442667e9bd72ed
paper_authors.avatarUrl: /avatars/6c0898227e49f69eded5c7d7ed0242b9.svg
paper_authors.fullname: Radu Florian
paper_authors.type: user
paper_authors.user: hansolosan
paper_authors._id: 65374541fa9f02750d26d08b
paper_authors.name: Salim Roukos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:51:39.449Z
paper_authors._id: 634d5b77fe861cc73a319aa2
paper_authors.avatarUrl: /avatars/963d60bd4346820547f0cf7f0da68457.svg
paper_authors.fullname: Salim Roukos
paper_authors.type: user
paper_authors.user: roukos
paper_authors._id: 65374541fa9f02750d26d08c
paper_authors.name: Ramón Fernandez Astudillo
paper_title: Ensemble-Instruct: Generating Instruction-Tuning Data with a
    Heterogeneous Mixture of LMs
paper_summary: Using in-context learning (ICL) for data generation, techniques such as
  Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023)
  can train strong conversational agents with only a small amount of human
  supervision. One limitation of these approaches is that they resort to very
  large language models (around 175B parameters) that are also proprietary and
  non-public. Here we explore the application of such techniques to language
  models that are much smaller (around 10B--40B parameters) and have permissive
  licenses. We find the Self-Instruct approach to be less effective at these
  sizes and propose new ICL methods that draw on two main ideas: (a)
  Categorization and simplification of the ICL templates to make prompt learning
  easier for the LM, and (b) Ensembling over multiple LM outputs to help select
  high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct
  seed tasks and employs separate pipelines for instructions that require an
  input and instructions that do not. Empirical investigations with different LMs
  show that: (1) Our proposed method yields higher-quality instruction tuning
  data than Self-Instruct, (2) It improves performances of both vanilla and
  instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned
  LMs generate more useful outputs than their larger un-tuned counterparts. Our
  codebase is available at https://github.com/IBM/ensemble-instruct.
paper_ai_summary: A new in-context learning method is proposed to generate high-quality instruction tuning data for smaller, open-access language models, improving their performance.
paper_ai_keywords: in-context learning ICL Self-Instruct Alpaca large language models small language models prompt learning ensembling instruction tuning codebase