title: Training Chain-of-Thought via Latent-Variable Inference
summary: Large language models (LLMs) solve problems more accurately and interpretably
  when instructed to work out the answer step by step using a
  ``chain-of-thought'' (CoT) prompt. One can also improve LLMs' performance on a
  specific task by supervised fine-tuning, i.e., by using gradient ascent on some
  tunable parameters to maximize the average log-likelihood of correct answers
  from a labeled training set. Naively combining CoT with supervised tuning
  requires supervision not just of the correct answers, but also of detailed
  rationales that lead to those answers; these rationales are expensive to
  produce by hand. Instead, we propose a fine-tuning strategy that tries to
  maximize the marginal log-likelihood of generating a correct answer
  using CoT prompting, approximately averaging over all possible rationales. The
  core challenge is sampling from the posterior over rationales conditioned on
  the correct answer; we address it using a simple Markov-chain Monte Carlo
  (MCMC) expectation-maximization (EM) algorithm inspired by the self-taught
  reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent
  contrastive divergence. This algorithm also admits a novel control-variate
  technique that drives the variance of our gradient estimates to zero as the
  model improves. Applying our technique to GSM8K and the tasks in BIG-Bench
  Hard, we find that this MCMC-EM fine-tuning technique typically improves the
  model's accuracy on held-out examples more than STaR or prompt-tuning with or
  without CoT.
paper_authors._id: 656fd424ee12f4a122a1078b
paper_authors.name: Du Phan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:10:40.064Z
paper_authors._id: 6451f8b8ff826ad744731702
paper_authors.avatarUrl: /avatars/1353d639ff24b06928eb139f86e9f03d.svg
paper_authors.fullname: Du Phan
paper_authors.type: user
paper_authors.user: duvinh
paper_authors._id: 656fd424ee12f4a122a1078c
paper_authors.name: Matthew D. Hoffman
paper_authors._id: 656fd424ee12f4a122a1078d
paper_authors.name: David Dohan
paper_authors._id: 656fd424ee12f4a122a1078e
paper_authors.name: Sholto Douglas
paper_authors._id: 656fd424ee12f4a122a1078f
paper_authors.name: Tuan Anh Le
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:11:27.673Z
paper_authors._id: 652f59a97a8c08f81e6bbf4c
paper_authors.avatarUrl: /avatars/bdbee2320c992617e1f2180ddabcc550.svg
paper_authors.fullname: Tuan Anh Le
paper_authors.type: user
paper_authors.user: anhlbt
paper_authors._id: 656fd424ee12f4a122a10790
paper_authors.name: Aaron Parisi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:11:54.116Z
paper_authors._id: 635f493ba81c7f7424a702ae
paper_authors.avatarUrl: /avatars/6f05d4f76609607966ed4635cb1e0a02.svg
paper_authors.fullname: Aaron Parisi
paper_authors.type: user
paper_authors.user: AnotherA
paper_authors._id: 656fd424ee12f4a122a10791
paper_authors.name: Pavel Sountsov
paper_authors._id: 656fd424ee12f4a122a10792
paper_authors.name: Charles Sutton
paper_authors._id: 656fd424ee12f4a122a10793
paper_authors.name: Sharad Vikram
paper_authors._id: 656fd424ee12f4a122a10794
paper_authors.name: Rif A. Saurous
paper_title: Training Chain-of-Thought via Latent-Variable Inference
paper_summary: Large language models (LLMs) solve problems more accurately and interpretably
  when instructed to work out the answer step by step using a
  ``chain-of-thought'' (CoT) prompt. One can also improve LLMs' performance on a
  specific task by supervised fine-tuning, i.e., by using gradient ascent on some
  tunable parameters to maximize the average log-likelihood of correct answers
  from a labeled training set. Naively combining CoT with supervised tuning
  requires supervision not just of the correct answers, but also of detailed
  rationales that lead to those answers; these rationales are expensive to
  produce by hand. Instead, we propose a fine-tuning strategy that tries to
  maximize the marginal log-likelihood of generating a correct answer
  using CoT prompting, approximately averaging over all possible rationales. The
  core challenge is sampling from the posterior over rationales conditioned on
  the correct answer; we address it using a simple Markov-chain Monte Carlo
  (MCMC) expectation-maximization (EM) algorithm inspired by the self-taught
  reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent
  contrastive divergence. This algorithm also admits a novel control-variate
  technique that drives the variance of our gradient estimates to zero as the
  model improves. Applying our technique to GSM8K and the tasks in BIG-Bench
  Hard, we find that this MCMC-EM fine-tuning technique typically improves the
  model's accuracy on held-out examples more than STaR or prompt-tuning with or
  without CoT.
paper_ai_summary: A fine-tuning strategy using MCMC-EM improves large language models' performance on specific tasks by maximizing the marginal log-likelihood of correct answers via chain-of-thought prompting without requiring detailed rationales.
paper_ai_keywords: chain-of-thought (CoT) large language models (LLMs) supervised fine-tuning gradient ascent log-likelihood marginal log-likelihood Markov-chain Monte Carlo (MCMC) expectation-maximization (EM) self-taught reasoner (STaR) memoized wake-sleep Markovian score climbing persistent contrastive divergence control-variate technique GSM8K BIG-Bench Hard