Thinking LLMs: General Instruction Following with Thought Generation
LLMs are typically trained to answer user questions or follow instructions
similarly to how human experts respond. However, in the standard alignment
framework they lack the basic ability of explicit thinking before answering.
Thinking is important for complex questions that require reasoning and planning
-- but can be applied to any task. We propose a training method for equipping
existing LLMs with such thinking abilities for general instruction following
without use of additional human data. We achieve this by an iterative search
and optimization procedure that explores the space of possible thought
generations, allowing the model to learn how to think without direct
supervision. For each instruction, the thought candidates are scored using a
judge model to evaluate their responses only, and then optimized via preference
optimization. We show that this procedure leads to superior performance on
AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning
categories such as marketing, health and general knowledge, in addition to more
traditional reasoning & problem-solving tasks.
670e8838ba29b3fca2222f69
Tianhao Wu
670e8838ba29b3fca2222f6a
Janice Lan
670e8838ba29b3fca2222f6b
Weizhe Yuan
admin_assigned
2024-10-15T16:45:50.569Z
6172aa78c8e66e2aa84c06b8
https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
Weizhe Yuan
user
weizhey
670e8838ba29b3fca2222f6c
Jiantao Jiao
admin_assigned
2024-10-15T16:45:56.366Z
653b306986b88947d5cacfa4
/avatars/21ebd4daf35ec67c7d5f9b0a53628b00.svg
Jiantao Jiao
user
nexus-jt-llm
670e8838ba29b3fca2222f6d
Jason Weston
admin_assigned
2024-10-15T16:46:02.044Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
670e8838ba29b3fca2222f6e
Sainbayar Sukhbaatar
admin_assigned
2024-10-15T16:46:07.400Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
Thinking LLMs: General Instruction Following with Thought Generation
LLMs are typically trained to answer user questions or follow instructions
similarly to how human experts respond. However, in the standard alignment
framework they lack the basic ability of explicit thinking before answering.
Thinking is important for complex questions that require reasoning and planning
-- but can be applied to any task. We propose a training method for equipping
existing LLMs with such thinking abilities for general instruction following
without use of additional human data. We achieve this by an iterative search
and optimization procedure that explores the space of possible thought
generations, allowing the model to learn how to think without direct
supervision. For each instruction, the thought candidates are scored using a
judge model to evaluate their responses only, and then optimized via preference
optimization. We show that this procedure leads to superior performance on
AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning
categories such as marketing, health and general knowledge, in addition to more
traditional reasoning & problem-solving tasks.
An iterative search and optimization procedure equips LLMs with thinking abilities for general instruction following, improving performance across various categories, including reasoning, marketing, health, and general knowledge.
LLMs instruction following explicit thinking reasoning planning iterative search optimization thought generations judge model preference optimization AlpacaEval Arena-Hard