title: Branch-Solve-Merge Improves Large Language Model Evaluation and
    Generation
summary: Large Language Models (LLMs) are frequently used for multi-faceted language
  generation and evaluation tasks that involve satisfying intricate user
  constraints or taking into account multiple aspects and criteria. However,
  their performance can fall short, due to the model's lack of coherence and
  inability to plan and decompose the problem. We propose Branch-Solve-Merge
  (BSM), a Large Language Model program (Schlag et al., 2023) for tackling such
  challenging natural language tasks. It consists of branch, solve, and merge
  modules that are parameterized with specific prompts to the base LLM. These
  three modules plan a decomposition of the task into multiple parallel
  sub-tasks, independently solve them, and fuse the solutions to the sub-tasks.
  We apply our method to the tasks of LLM response evaluation and constrained
  text generation and evaluate its effectiveness with multiple LLMs, including
  Vicuna, LLaMA-2-chat, and GPT-4. BSM improves the evaluation correctness and
  consistency for each LLM by enhancing human-LLM agreement by up to 26%,
  reducing length and pairwise position biases by up to 50%, and allowing
  LLaMA-2-chat to match or outperform GPT-4 on most domains. On the constraint
  story generation task, BSM improves the coherence of the stories while also
  improving constraint satisfaction by 12%.
paper_authors._id: 65373db1d94ca568059aa723
paper_authors.name: Swarnadeep Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:59:18.382Z
paper_authors._id: 64b6feee17681d64b19b112b
paper_authors.avatarUrl: /avatars/afe708392d16af5b55d3bc4b42a585e8.svg
paper_authors.fullname: Swarnadeep Saha
paper_authors.type: user
paper_authors.user: swarnaNLP
paper_authors._id: 65373db1d94ca568059aa724
paper_authors.name: Omer Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:59:24.076Z
paper_authors._id: 610b71d7f5dd7bcefedaff55
paper_authors.avatarUrl: /avatars/b489c021e906010e661313da0b4aafab.svg
paper_authors.fullname: Omer Levy
paper_authors.type: user
paper_authors.user: omerlevy
paper_authors._id: 65373db1d94ca568059aa725
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 65373db1d94ca568059aa726
paper_authors.name: Mohit Bansal
paper_authors._id: 65373db1d94ca568059aa727
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T09:00:09.336Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 65373db1d94ca568059aa728
paper_authors.name: Xian Li
paper_title: Branch-Solve-Merge Improves Large Language Model Evaluation and
    Generation
paper_summary: Large Language Models (LLMs) are frequently used for multi-faceted language
  generation and evaluation tasks that involve satisfying intricate user
  constraints or taking into account multiple aspects and criteria. However,
  their performance can fall short, due to the model's lack of coherence and
  inability to plan and decompose the problem. We propose Branch-Solve-Merge
  (BSM), a Large Language Model program (Schlag et al., 2023) for tackling such
  challenging natural language tasks. It consists of branch, solve, and merge
  modules that are parameterized with specific prompts to the base LLM. These
  three modules plan a decomposition of the task into multiple parallel
  sub-tasks, independently solve them, and fuse the solutions to the sub-tasks.
  We apply our method to the tasks of LLM response evaluation and constrained
  text generation and evaluate its effectiveness with multiple LLMs, including
  Vicuna, LLaMA-2-chat, and GPT-4. BSM improves the evaluation correctness and
  consistency for each LLM by enhancing human-LLM agreement by up to 26%,
  reducing length and pairwise position biases by up to 50%, and allowing
  LLaMA-2-chat to match or outperform GPT-4 on most domains. On the constraint
  story generation task, BSM improves the coherence of the stories while also
  improving constraint satisfaction by 12%.
paper_ai_keywords: Large Language Models (LLMs) Branch-Solve-Merge (BSM) parameterized prompts branch module solve module merge module task decomposition LLM response evaluation constrained text generation LLM decoders Vicuna LLaMA-2-chat GPT-4 evaluation correctness consistency human-LLM evaluation length bias pairwise position bias constraint story generation coherence of stories constraint satisfaction