title: The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
summary: Entropy minimization (EM) trains the model to concentrate even more
  probability mass on its most confident outputs. We show that this simple
  objective alone, without any labeled data, can substantially improve large
  language models' (LLMs) performance on challenging math, physics, and coding
  tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy
  similarly to instruction finetuning, but on unlabeled outputs drawn from the
  model; (2) EM-RL: reinforcement learning with negative entropy as the only
  reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce
  entropy without any training data or parameter updates. On Qwen-7B, EM-RL,
  without any labeled data, achieves comparable or better performance than strong
  RL baselines such as GRPO and RLOO that are trained on 60K labeled examples.
  Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of
  proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the
  challenging SciCode benchmark, while being 3x more efficient than
  self-consistency and sequential refinement. Our findings reveal that many
  pretrained LLMs possess previously underappreciated reasoning capabilities that
  can be effectively elicited through entropy minimization alone, without any
  labeled data or even any parameter updates.
paper_authors._id: 682f8212bcb3f2f6c7bfe588
paper_authors.name: Shivam Agarwal
paper_authors._id: 682f8212bcb3f2f6c7bfe589
paper_authors.name: Zimin Zhang
paper_authors._id: 682f8212bcb3f2f6c7bfe58a
paper_authors.name: Lifan Yuan
paper_authors._id: 682f8212bcb3f2f6c7bfe58b
paper_authors.name: Jiawei Han
paper_authors._id: 682f8212bcb3f2f6c7bfe58c
paper_authors.name: Hao Peng
paper_title: The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
paper_summary: Entropy minimization (EM) trains the model to concentrate even more
  probability mass on its most confident outputs. We show that this simple
  objective alone, without any labeled data, can substantially improve large
  language models' (LLMs) performance on challenging math, physics, and coding
  tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy
  similarly to instruction finetuning, but on unlabeled outputs drawn from the
  model; (2) EM-RL: reinforcement learning with negative entropy as the only
  reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce
  entropy without any training data or parameter updates. On Qwen-7B, EM-RL,
  without any labeled data, achieves comparable or better performance than strong
  RL baselines such as GRPO and RLOO that are trained on 60K labeled examples.
  Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of
  proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the
  challenging SciCode benchmark, while being 3x more efficient than
  self-consistency and sequential refinement. Our findings reveal that many
  pretrained LLMs possess previously underappreciated reasoning capabilities that
  can be effectively elicited through entropy minimization alone, without any
  labeled data or even any parameter updates.
paper_ai_summary: Entropy minimization improves large language models' performance on math, physics, and coding tasks without labeled data through various methods like fine-tuning, reinforcement learning, and inference-time adjustments.
paper_ai_keywords: entropy minimization EM-FT EM-RL EM-INF token-level entropy instruction finetuning reinforcement learning logit adjustment Qwen-7B Qwen-32B SciCode benchmark self-consistency sequential refinement large language models