title: Thinking LLMs: General Instruction Following with Thought Generation
summary: LLMs are typically trained to answer user questions or follow instructions
  similarly to how human experts respond. However, in the standard alignment
  framework they lack the basic ability of explicit thinking before answering.
  Thinking is important for complex questions that require reasoning and planning
  -- but can be applied to any task. We propose a training method for equipping
  existing LLMs with such thinking abilities for general instruction following
  without use of additional human data. We achieve this by an iterative search
  and optimization procedure that explores the space of possible thought
  generations, allowing the model to learn how to think without direct
  supervision. For each instruction, the thought candidates are scored using a
  judge model to evaluate their responses only, and then optimized via preference
  optimization. We show that this procedure leads to superior performance on
  AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning
  categories such as marketing, health and general knowledge, in addition to more
  traditional reasoning & problem-solving tasks.
paper_authors._id: 670e8838ba29b3fca2222f69
paper_authors.name: Tianhao Wu
paper_authors._id: 670e8838ba29b3fca2222f6a
paper_authors.name: Janice Lan
paper_authors._id: 670e8838ba29b3fca2222f6b
paper_authors.name: Weizhe Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:45:50.569Z
paper_authors._id: 6172aa78c8e66e2aa84c06b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
paper_authors.fullname: Weizhe Yuan
paper_authors.type: user
paper_authors.user: weizhey
paper_authors._id: 670e8838ba29b3fca2222f6c
paper_authors.name: Jiantao Jiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:45:56.366Z
paper_authors._id: 653b306986b88947d5cacfa4
paper_authors.avatarUrl: /avatars/21ebd4daf35ec67c7d5f9b0a53628b00.svg
paper_authors.fullname: Jiantao Jiao
paper_authors.type: user
paper_authors.user: nexus-jt-llm
paper_authors._id: 670e8838ba29b3fca2222f6d
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:46:02.044Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 670e8838ba29b3fca2222f6e
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:46:07.400Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_title: Thinking LLMs: General Instruction Following with Thought Generation
paper_summary: LLMs are typically trained to answer user questions or follow instructions
  similarly to how human experts respond. However, in the standard alignment
  framework they lack the basic ability of explicit thinking before answering.
  Thinking is important for complex questions that require reasoning and planning
  -- but can be applied to any task. We propose a training method for equipping
  existing LLMs with such thinking abilities for general instruction following
  without use of additional human data. We achieve this by an iterative search
  and optimization procedure that explores the space of possible thought
  generations, allowing the model to learn how to think without direct
  supervision. For each instruction, the thought candidates are scored using a
  judge model to evaluate their responses only, and then optimized via preference
  optimization. We show that this procedure leads to superior performance on
  AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning
  categories such as marketing, health and general knowledge, in addition to more
  traditional reasoning & problem-solving tasks.
paper_ai_summary: An iterative search and optimization procedure equips LLMs with thinking abilities for general instruction following, improving performance across various categories, including reasoning, marketing, health, and general knowledge.
paper_ai_keywords: LLMs instruction following explicit thinking reasoning planning iterative search optimization thought generations judge model preference optimization AlpacaEval Arena-Hard