title: Auto-Instruct: Automatic Instruction Generation and Ranking for
    Black-Box Language Models
summary: Large language models (LLMs) can perform a wide range of tasks by following
  natural language instructions, without the necessity of task-specific
  fine-tuning. Unfortunately, the performance of LLMs is greatly influenced by
  the quality of these instructions, and manually writing effective instructions
  for each task is a laborious and subjective process. In this paper, we
  introduce Auto-Instruct, a novel method to automatically improve the quality of
  instructions provided to LLMs. Our method leverages the inherent generative
  ability of LLMs to produce diverse candidate instructions for a given task, and
  then ranks them using a scoring model trained on a variety of 575 existing NLP
  tasks. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both
  human-written instructions and existing baselines of LLM-generated
  instructions. Furthermore, our method exhibits notable generalizability even
  with other LLMs that are not incorporated into its training process.
paper_authors._id: 6535e7cea2c81a3d29b5e2d5
paper_authors.name: Zhihan Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-23T03:42:51.151Z
paper_authors._id: 63bf9695da08ed054400205e
paper_authors.avatarUrl: /avatars/b6fca49559a61cf66628088c60d26c10.svg
paper_authors.fullname: Zhihan Zhang
paper_authors.type: user
paper_authors.user: zhihz0535
paper_authors._id: 6535e7cea2c81a3d29b5e2d6
paper_authors.name: Shuohang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:04:38.751Z
paper_authors._id: 6463b2247572c66a8e625a57
paper_authors.avatarUrl: /avatars/7722fb5649d42d966ce1e478946d5f8f.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Shuohang
paper_authors._id: 6535e7cea2c81a3d29b5e2d7
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:05:28.503Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 6535e7cea2c81a3d29b5e2d8
paper_authors.name: Yichong Xu
paper_authors._id: 6535e7cea2c81a3d29b5e2d9
paper_authors.name: Dan Iter
paper_authors._id: 6535e7cea2c81a3d29b5e2da
paper_authors.name: Qingkai Zeng
paper_authors._id: 6535e7cea2c81a3d29b5e2db
paper_authors.name: Yang Liu
paper_authors._id: 6535e7cea2c81a3d29b5e2dc
paper_authors.name: Chenguang Zhu
paper_authors._id: 6535e7cea2c81a3d29b5e2dd
paper_authors.name: Meng Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:51.012Z
paper_authors._id: 62cdc931c589e4a9e2411d39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
paper_authors.fullname: Meng Jiang
paper_authors.type: user
paper_authors.user: mjiang89
paper_title: Auto-Instruct: Automatic Instruction Generation and Ranking for
    Black-Box Language Models
paper_summary: Large language models (LLMs) can perform a wide range of tasks by following
  natural language instructions, without the necessity of task-specific
  fine-tuning. Unfortunately, the performance of LLMs is greatly influenced by
  the quality of these instructions, and manually writing effective instructions
  for each task is a laborious and subjective process. In this paper, we
  introduce Auto-Instruct, a novel method to automatically improve the quality of
  instructions provided to LLMs. Our method leverages the inherent generative
  ability of LLMs to produce diverse candidate instructions for a given task, and
  then ranks them using a scoring model trained on a variety of 575 existing NLP
  tasks. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both
  human-written instructions and existing baselines of LLM-generated
  instructions. Furthermore, our method exhibits notable generalizability even
  with other LLMs that are not incorporated into its training process.
paper_ai_keywords: large language models (LLMs) task-specific fine-tuning Auto-Instruct scoring model NLP tasks out-of-domain tasks generative ability candidate instructions