title: Self-Consistency Preference Optimization
summary: Self-alignment, whereby models learn to improve themselves without human
  annotation, is a rapidly growing research area. However, existing techniques
  often fail to improve complex reasoning tasks due to the difficulty of
  assigning correct rewards. An orthogonal approach that is known to improve
  correctness is self-consistency, a method applied at inference time based on
  multiple sampling in order to find the most consistent answer. In this work, we
  extend the self-consistency concept to help train models. We thus introduce
  self-consistency preference optimization (ScPO), which iteratively trains
  consistent answers to be preferred over inconsistent ones on unsupervised new
  problems. We show ScPO leads to large improvements over conventional reward
  model training on reasoning tasks such as GSM8K and MATH, closing the gap with
  supervised training with gold answers or preferences, and that combining ScPO
  with standard supervised learning improves results even further. On ZebraLogic,
  ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and
  Claude-3 Haiku.
paper_authors._id: 672cbe66a1e855c3297c98d4
paper_authors.name: Archiki Prasad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:33:11.771Z
paper_authors._id: 607aeae5d2cd8c150e6ae074
paper_authors.avatarUrl: /avatars/a087743b98b6fe2181283a9610db4ec4.svg
paper_authors.fullname: Archiki Prasad
paper_authors.type: user
paper_authors.user: archiki
paper_authors._id: 672cbe66a1e855c3297c98d5
paper_authors.name: Weizhe Yuan
paper_authors._id: 672cbe66a1e855c3297c98d6
paper_authors.name: Richard Yuanzhe Pang
paper_authors._id: 672cbe66a1e855c3297c98d7
paper_authors.name: Jing Xu
paper_authors._id: 672cbe66a1e855c3297c98d8
paper_authors.name: Maryam Fazel-Zarandi
paper_authors._id: 672cbe66a1e855c3297c98d9
paper_authors.name: Mohit Bansal
paper_authors._id: 672cbe66a1e855c3297c98da
paper_authors.name: Sainbayar Sukhbaatar
paper_authors._id: 672cbe66a1e855c3297c98db
paper_authors.name: Jason Weston
paper_authors._id: 672cbe66a1e855c3297c98dc
paper_authors.name: Jane Yu
paper_title: Self-Consistency Preference Optimization
paper_summary: Self-alignment, whereby models learn to improve themselves without human
  annotation, is a rapidly growing research area. However, existing techniques
  often fail to improve complex reasoning tasks due to the difficulty of
  assigning correct rewards. An orthogonal approach that is known to improve
  correctness is self-consistency, a method applied at inference time based on
  multiple sampling in order to find the most consistent answer. In this work, we
  extend the self-consistency concept to help train models. We thus introduce
  self-consistency preference optimization (ScPO), which iteratively trains
  consistent answers to be preferred over inconsistent ones on unsupervised new
  problems. We show ScPO leads to large improvements over conventional reward
  model training on reasoning tasks such as GSM8K and MATH, closing the gap with
  supervised training with gold answers or preferences, and that combining ScPO
  with standard supervised learning improves results even further. On ZebraLogic,
  ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and
  Claude-3 Haiku.
paper_ai_keywords: self-alignment self-consistency reward model training self-consistency preference optimization (ScPO) GSM8K MATH reasoning tasks ZebraLogic Llama-3 8B Llama-3 70B Gemma-2 27B Claude-3 Haiku