title: Large Language Model Cascades with Mixture of Thoughts Representations
    for Cost-efficient Reasoning
summary: Large language models (LLMs) such as GPT-4 have exhibited remarkable
  performance in a variety of tasks, but this strong performance often comes with
  the high expense of using paid API services. In this paper, we are motivated to
  study building an LLM cascade to save the cost of using LLMs, particularly for
  performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline
  follows the intuition that simpler questions can be addressed by a weaker but
  more affordable LLM, whereas only the challenging questions necessitate the
  stronger and more expensive LLM. To realize this decision-making, we consider
  the "answer consistency" of the weaker LLM as a signal of the question
  difficulty and propose several methods for the answer sampling and consistency
  checking, including one leveraging a mixture of two thought representations
  (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six
  reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and
  stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can
  achieve performance comparable to using solely the stronger LLM but require
  only 40% of its cost.
paper_authors._id: 651f611382dbe82d3f63b631
paper_authors.name: Murong Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:57:41.787Z
paper_authors._id: 63e1ac9f7fbb6ae4d4f312a3
paper_authors.avatarUrl: /avatars/7d999543d3ca3df3fc8bb96cd3a3f8cb.svg
paper_authors.fullname: Murong Yue
paper_authors.type: user
paper_authors.user: MurongYue
paper_authors._id: 651f611382dbe82d3f63b632
paper_authors.name: Jie Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-10T08:17:34.298Z
paper_authors._id: 652486d606e839c5540af4d3
paper_authors.avatarUrl: /avatars/035acfaa1b6224853da0aba60d1bf0f7.svg
paper_authors.fullname: Jie Zhao
paper_authors.type: user
paper_authors.user: jie-zhao
paper_authors._id: 651f611382dbe82d3f63b633
paper_authors.name: Min Zhang
paper_authors._id: 651f611382dbe82d3f63b634
paper_authors.name: Liang Du
paper_authors._id: 651f611382dbe82d3f63b635
paper_authors.name: Ziyu Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:58:52.259Z
paper_authors._id: 62d87e143cd901df9446a37a
paper_authors.avatarUrl: /avatars/b9a7d24bbe6e49ac90902fafee3a74e7.svg
paper_authors.fullname: Ziyu Yao
paper_authors.type: user
paper_authors.user: ziyuyao
paper_title: Large Language Model Cascades with Mixture of Thoughts Representations
    for Cost-efficient Reasoning
paper_summary: Large language models (LLMs) such as GPT-4 have exhibited remarkable
  performance in a variety of tasks, but this strong performance often comes with
  the high expense of using paid API services. In this paper, we are motivated to
  study building an LLM cascade to save the cost of using LLMs, particularly for
  performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline
  follows the intuition that simpler questions can be addressed by a weaker but
  more affordable LLM, whereas only the challenging questions necessitate the
  stronger and more expensive LLM. To realize this decision-making, we consider
  the "answer consistency" of the weaker LLM as a signal of the question
  difficulty and propose several methods for the answer sampling and consistency
  checking, including one leveraging a mixture of two thought representations
  (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six
  reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and
  stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can
  achieve performance comparable to using solely the stronger LLM but require
  only 40% of its cost.
paper_ai_summary: An LLM cascade system reduces costs by using a less expensive LLM for simpler tasks and a more expensive LLM for challenging tasks, maintaining performance through answer consistency checks.
paper_ai_keywords: LLM cascade answer consistency Chain-of-Thought Program-of-Thought reasoning benchmark datasets