title: InstructZero: Efficient Instruction Optimization for Black-Box Large
    Language Models
summary: Large language models~(LLMs) are instruction followers, but it can be
  challenging to find the best instruction for different situations, especially
  for black-box LLMs on which backpropagation is forbidden. Instead of directly
  optimizing the discrete instruction, we optimize a low-dimensional soft prompt
  applied to an open-source LLM to generate the instruction for the black-box
  LLM. On each iteration of the proposed method, which we call InstructZero, a
  soft prompt is converted into an instruction using the open-source LLM, which
  is then submitted to the black-box LLM for zero-shot evaluation, and the
  performance is sent to Bayesian optimization to produce new soft prompts
  improving the zero-shot performance. We evaluate InstructZero on different
  combinations of open-source LLMs and APIs including Vicuna and ChatGPT. Our
  results show that InstructZero outperforms SOTA auto-instruction methods across
  a variety of downstream tasks. Our code and data are publicly available at
  https://github.com/Lichang-Chen/InstructZero.
paper_authors._id: 647e877fe4d52fe0e0210f88
paper_authors.name: Lichang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T17:23:06.023Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 647e877fe4d52fe0e0210f89
paper_authors.name: Jiuhai Chen
paper_authors._id: 647e877fe4d52fe0e0210f8a
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-06T09:59:12.898Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_authors._id: 647e877fe4d52fe0e0210f8b
paper_authors.name: Heng Huang
paper_authors._id: 647e877fe4d52fe0e0210f8c
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T17:23:10.958Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: InstructZero: Efficient Instruction Optimization for Black-Box Large
    Language Models
paper_summary: Large language models~(LLMs) are instruction followers, but it can be
  challenging to find the best instruction for different situations, especially
  for black-box LLMs on which backpropagation is forbidden. Instead of directly
  optimizing the discrete instruction, we optimize a low-dimensional soft prompt
  applied to an open-source LLM to generate the instruction for the black-box
  LLM. On each iteration of the proposed method, which we call InstructZero, a
  soft prompt is converted into an instruction using the open-source LLM, which
  is then submitted to the black-box LLM for zero-shot evaluation, and the
  performance is sent to Bayesian optimization to produce new soft prompts
  improving the zero-shot performance. We evaluate InstructZero on different
  combinations of open-source LLMs and APIs including Vicuna and ChatGPT. Our
  results show that InstructZero outperforms SOTA auto-instruction methods across
  a variety of downstream tasks. Our code and data are publicly available at
  https://github.com/Lichang-Chen/InstructZero.
paper_ai_keywords: large language models (LLMs) soft prompt zero-shot evaluation Bayesian optimization Vicuna ChatGPT auto-instruction methods downstream tasks