title: Prompt Engineering a Prompt Engineer
summary: Prompt engineering is a challenging yet crucial task for optimizing the
  performance of large language models (LLMs). It requires complex reasoning to
  examine the model's errors, hypothesize what is missing or misleading in the
  current prompt, and communicate the task with clarity. While recent works
  indicate that LLMs can be meta-prompted to perform automatic prompt
  engineering, their potentials may not be fully untapped due to the lack of
  sufficient guidance to elicit complex reasoning capabilities in LLMs in the
  meta-prompt. In this work, we investigate the problem of "prompt engineering a
  prompt engineer" -- constructing a meta-prompt that more effectively guides
  LLMs to perform automatic prompt engineering. We introduce and analyze key
  components, such as a step-by-step reasoning template and context
  specification, which lead to improved performance. In addition, inspired by
  common optimization concepts such as batch size, step size and momentum, we
  introduce their verbalized counterparts to the meta-prompt and investigate
  their effects. Our final method, named PE2, finds a prompt that outperforms
  "let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
  GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
  Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
  industrial prompt. In these settings, PE2 achieves strong performance and
  outperforms prior automatic prompt engineering baselines. Further, we show that
  PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
  prompts, and presents non-trivial counterfactual reasoning abilities.
paper_authors._id: 65519961685ba4c13dae2616
paper_authors.name: Qinyuan Ye
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-11T22:13:36.656Z
paper_authors._id: 64a3a76868fe7eed8fc1a72c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a3a76868fe7eed8fc1a72c/1aNwBBt395r7PHB5yCiW2.jpeg
paper_authors.fullname: Qinyuan Ye
paper_authors.type: user
paper_authors.user: qinyuany
paper_authors._id: 65519961685ba4c13dae2617
paper_authors.name: Maxamed Axmed
paper_authors._id: 65519961685ba4c13dae2618
paper_authors.name: Reid Pryzant
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:44:40.445Z
paper_authors._id: 64e3d38c618cd90997ee0138
paper_authors.avatarUrl: /avatars/847210a609b156fdb9d824cc815bde27.svg
paper_authors.fullname: Reid Pryzant
paper_authors.type: user
paper_authors.user: rpryzant
paper_authors._id: 65519961685ba4c13dae2619
paper_authors.name: Fereshte Khani
paper_title: Prompt Engineering a Prompt Engineer
paper_summary: Prompt engineering is a challenging yet crucial task for optimizing the
  performance of large language models (LLMs). It requires complex reasoning to
  examine the model's errors, hypothesize what is missing or misleading in the
  current prompt, and communicate the task with clarity. While recent works
  indicate that LLMs can be meta-prompted to perform automatic prompt
  engineering, their potentials may not be fully untapped due to the lack of
  sufficient guidance to elicit complex reasoning capabilities in LLMs in the
  meta-prompt. In this work, we investigate the problem of "prompt engineering a
  prompt engineer" -- constructing a meta-prompt that more effectively guides
  LLMs to perform automatic prompt engineering. We introduce and analyze key
  components, such as a step-by-step reasoning template and context
  specification, which lead to improved performance. In addition, inspired by
  common optimization concepts such as batch size, step size and momentum, we
  introduce their verbalized counterparts to the meta-prompt and investigate
  their effects. Our final method, named PE2, finds a prompt that outperforms
  "let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
  GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
  Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
  industrial prompt. In these settings, PE2 achieves strong performance and
  outperforms prior automatic prompt engineering baselines. Further, we show that
  PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
  prompts, and presents non-trivial counterfactual reasoning abilities.
paper_ai_summary: The study develops a meta-prompting technique, PE2, to enhance automatic prompt engineering for large language models, demonstrating improved performance and sophisticated reasoning capabilities across various datasets and tasks.
paper_ai_keywords: large language models prompt engineering meta-prompting step-by-step reasoning template context specification batch size step size momentum MultiArith dataset GSM8K dataset Instruction Induction benchmark counterfactual tasks industrial prompt