title: Divide-or-Conquer? Which Part Should You Distill Your LLM?
summary: Recent methods have demonstrated that Large Language Models (LLMs) can solve
  reasoning tasks better when they are encouraged to solve subtasks of the main
  task first. In this paper we devise a similar strategy that breaks down
  reasoning tasks into a problem decomposition phase and a problem solving phase
  and show that the strategy is able to outperform a single stage solution.
  Further, we hypothesize that the decomposition should be easier to distill into
  a smaller model compared to the problem solving because the latter requires
  large amounts of domain knowledge while the former only requires learning
  general problem solving strategies. We propose methods to distill these two
  capabilities and evaluate their impact on reasoning outcomes and inference
  cost. We find that we can distill the problem decomposition phase and at the
  same time achieve good generalization across tasks, datasets, and models.
  However, it is harder to distill the problem solving capability without losing
  performance and the resulting distilled model struggles with generalization.
  These results indicate that by using smaller, distilled problem decomposition
  models in combination with problem solving LLMs we can achieve reasoning with
  cost-efficient inference and local adaptation.
paper_authors._id: 65dc01a3bda21d181bf6ea3d
paper_authors.name: Zhuofeng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-26T21:09:43.758Z
paper_authors._id: 63c75671656e7822e23df874
paper_authors.avatarUrl: /avatars/6fbe669287405b2b47cafcf4961cfa75.svg
paper_authors.fullname: Zhuofeng Wu
paper_authors.type: user
paper_authors.user: cserxy
paper_authors._id: 65dc01a3bda21d181bf6ea3e
paper_authors.name: He Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-26T21:09:46.303Z
paper_authors._id: 5f024bd8ff69163f6f59e775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
paper_authors.fullname: Richard He Bai
paper_authors.type: user
paper_authors.user: richardbaihe
paper_authors._id: 65dc01a3bda21d181bf6ea3f
paper_authors.name: Aonan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-16T10:37:22.872Z
paper_authors._id: 65f3e68a138c6ab771434e2d
paper_authors.avatarUrl: /avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
paper_authors.fullname: Aonan Zhang
paper_authors.type: user
paper_authors.user: AonanZhang
paper_authors._id: 65dc01a3bda21d181bf6ea40
paper_authors.name: Jiatao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T11:13:07.272Z
paper_authors._id: 6164e72d73996c363c52e66d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
paper_authors.fullname: Jiatao Gu
paper_authors.type: user
paper_authors.user: thomagram
paper_authors._id: 65dc01a3bda21d181bf6ea41
paper_authors.name: VG Vinod Vydiswaran
paper_authors._id: 65dc01a3bda21d181bf6ea42
paper_authors.name: Navdeep Jaitly
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T11:13:18.818Z
paper_authors._id: 631119df88942700629f4aab
paper_authors.avatarUrl: /avatars/251c20a8fd48831a8c0cc0d198a42455.svg
paper_authors.fullname: Navdeep Jaitly
paper_authors.type: user
paper_authors.user: ndjaitly
paper_authors._id: 65dc01a3bda21d181bf6ea43
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T11:14:09.471Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_title: Divide-or-Conquer? Which Part Should You Distill Your LLM?
paper_summary: Recent methods have demonstrated that Large Language Models (LLMs) can solve
  reasoning tasks better when they are encouraged to solve subtasks of the main
  task first. In this paper we devise a similar strategy that breaks down
  reasoning tasks into a problem decomposition phase and a problem solving phase
  and show that the strategy is able to outperform a single stage solution.
  Further, we hypothesize that the decomposition should be easier to distill into
  a smaller model compared to the problem solving because the latter requires
  large amounts of domain knowledge while the former only requires learning
  general problem solving strategies. We propose methods to distill these two
  capabilities and evaluate their impact on reasoning outcomes and inference
  cost. We find that we can distill the problem decomposition phase and at the
  same time achieve good generalization across tasks, datasets, and models.
  However, it is harder to distill the problem solving capability without losing
  performance and the resulting distilled model struggles with generalization.
  These results indicate that by using smaller, distilled problem decomposition
  models in combination with problem solving LLMs we can achieve reasoning with
  cost-efficient inference and local adaptation.
paper_ai_summary: A strategy breaking reasoning tasks into decomposition and solving phases allows distillation of the former for efficient inference and generalization.
paper_ai_keywords: Large Language Models subtasks problem decomposition phase problem solving phase single stage solution distilled model domain knowledge general problem solving strategies