Auto-Instruct: Automatic Instruction Generation and Ranking for
  Black-Box Language Models
Large language models (LLMs) can perform a wide range of tasks by following
natural language instructions, without the necessity of task-specific
fine-tuning. Unfortunately, the performance of LLMs is greatly influenced by
the quality of these instructions, and manually writing effective instructions
for each task is a laborious and subjective process. In this paper, we
introduce Auto-Instruct, a novel method to automatically improve the quality of
instructions provided to LLMs. Our method leverages the inherent generative
ability of LLMs to produce diverse candidate instructions for a given task, and
then ranks them using a scoring model trained on a variety of 575 existing NLP
tasks. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both
human-written instructions and existing baselines of LLM-generated
instructions. Furthermore, our method exhibits notable generalizability even
with other LLMs that are not incorporated into its training process.
6535e7cea2c81a3d29b5e2d5
Zhihan Zhang
extracted_confirmed
2023-10-23T03:42:51.151Z
63bf9695da08ed054400205e
/avatars/5223b212431337c6186db96ac70b7ae7.svg
Zhihan Zhang
user
zhihz0535
6535e7cea2c81a3d29b5e2d6
Shuohang Wang
admin_assigned
2023-10-23T09:04:38.751Z
6463b2247572c66a8e625a57
/avatars/7722fb5649d42d966ce1e478946d5f8f.svg
Wang
user
Shuohang
6535e7cea2c81a3d29b5e2d7
Wenhao Yu
admin_assigned
2023-10-23T09:05:28.503Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
6535e7cea2c81a3d29b5e2d8
Yichong Xu
6535e7cea2c81a3d29b5e2d9
Dan Iter
6535e7cea2c81a3d29b5e2da
Qingkai Zeng
6535e7cea2c81a3d29b5e2db
Yang Liu
6535e7cea2c81a3d29b5e2dc
Chenguang Zhu
6535e7cea2c81a3d29b5e2dd
Meng Jiang
claimed_verified
2023-10-24T07:29:51.012Z
62cdc931c589e4a9e2411d39
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdc931c589e4a9e2411d39/Q91Xp6WeI50AjViMzbD3Q.png
Meng Jiang
user
mjiang89
Auto-Instruct: Automatic Instruction Generation and Ranking for
  Black-Box Language Models
Large language models (LLMs) can perform a wide range of tasks by following
natural language instructions, without the necessity of task-specific
fine-tuning. Unfortunately, the performance of LLMs is greatly influenced by
the quality of these instructions, and manually writing effective instructions
for each task is a laborious and subjective process. In this paper, we
introduce Auto-Instruct, a novel method to automatically improve the quality of
instructions provided to LLMs. Our method leverages the inherent generative
ability of LLMs to produce diverse candidate instructions for a given task, and
then ranks them using a scoring model trained on a variety of 575 existing NLP
tasks. In experiments on 118 out-of-domain tasks, Auto-Instruct surpasses both
human-written instructions and existing baselines of LLM-generated
instructions. Furthermore, our method exhibits notable generalizability even
with other LLMs that are not incorporated into its training process.
Auto-Instruct automatically improves the quality of instructions for large language models by generating and ranking diverse candidates using a scoring model trained on various NLP tasks.
large language models (LLMs) natural language instructions task-specific fine-tuning generative ability scoring model NLP tasks out-of-domain tasks human-written instructions LLM-generated instructions generalizability