title: Beyond Human Data: Scaling Self-Training for Problem-Solving with
    Language Models
summary: Fine-tuning language models~(LMs) on human-generated data remains a prevalent
  practice. However, the performance of such models is often limited by the
  quantity and diversity of high-quality human data. In this paper, we explore
  whether we can go beyond human data on tasks where we have access to scalar
  feedback, for example, on math problems where one can verify correctness. To do
  so, we investigate a simple self-training method based on
  expectation-maximization, which we call ReST^{EM}, where we (1) generate
  samples from the model and filter them using binary feedback, (2) fine-tune the
  model on these samples, and (3) repeat this process a few times. Testing on
  advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find
  that ReST^{EM} scales favorably with model size and significantly surpasses
  fine-tuning only on human data. Overall, our findings suggest self-training
  with feedback can substantially reduce dependence on human-generated data.
paper_authors._id: 6577f1e4ee33d547aebf898d
paper_authors.name: Avi Singh
paper_authors._id: 6577f1e4ee33d547aebf898e
paper_authors.name: John D. Co-Reyes
paper_authors._id: 6577f1e4ee33d547aebf898f
paper_authors.name: Rishabh Agarwal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-12T05:38:44.332Z
paper_authors._id: 647875e37b370854241b89a8
paper_authors.avatarUrl: /avatars/fd25f7e5960e4e132b9ee689f114820b.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: agarwl
paper_authors._id: 6577f1e4ee33d547aebf8990
paper_authors.name: Ankesh Anand
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:23:09.319Z
paper_authors._id: 626321a613cc9ec91bd7de48
paper_authors.avatarUrl: /avatars/79e4f1843a5a8041197b01772f21bc48.svg
paper_authors.fullname: Ankesh Anand
paper_authors.type: user
paper_authors.user: ankeshanand
paper_authors._id: 6577f1e4ee33d547aebf8991
paper_authors.name: Piyush Patil
paper_authors._id: 6577f1e4ee33d547aebf8992
paper_authors.name: Peter J. Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:23:45.836Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_authors._id: 6577f1e4ee33d547aebf8993
paper_authors.name: James Harrison
paper_authors._id: 6577f1e4ee33d547aebf8994
paper_authors.name: Jaehoon Lee
paper_authors._id: 6577f1e4ee33d547aebf8995
paper_authors.name: Kelvin Xu
paper_authors._id: 6577f1e4ee33d547aebf8996
paper_authors.name: Aaron Parisi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:25:14.232Z
paper_authors._id: 635f493ba81c7f7424a702ae
paper_authors.avatarUrl: /avatars/6f05d4f76609607966ed4635cb1e0a02.svg
paper_authors.fullname: Aaron Parisi
paper_authors.type: user
paper_authors.user: AnotherA
paper_authors._id: 6577f1e4ee33d547aebf8997
paper_authors.name: Abhishek Kumar
paper_authors._id: 6577f1e4ee33d547aebf8998
paper_authors.name: Alex Alemi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:25:41.874Z
paper_authors._id: 6303ff84fc783bfc74441afc
paper_authors.avatarUrl: /avatars/bc295e0d94c9c8be0c6a8264587ad4a5.svg
paper_authors.fullname: Alex Alemi
paper_authors.type: user
paper_authors.user: knowflake
paper_authors._id: 6577f1e4ee33d547aebf8999
paper_authors.name: Alex Rizkowsky
paper_authors._id: 6577f1e4ee33d547aebf899a
paper_authors.name: Azade Nova
paper_authors._id: 6577f1e4ee33d547aebf899b
paper_authors.name: Ben Adlam
paper_authors._id: 6577f1e4ee33d547aebf899c
paper_authors.name: Bernd Bohnet
paper_authors._id: 6577f1e4ee33d547aebf899d
paper_authors.name: Hanie Sedghi
paper_authors._id: 6577f1e4ee33d547aebf899e
paper_authors.name: Igor Mordatch
paper_authors._id: 6577f1e4ee33d547aebf899f
paper_authors.name: Isabelle Simpson
paper_authors._id: 6577f1e4ee33d547aebf89a0
paper_authors.name: Izzeddin Gur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:26:12.768Z
paper_authors._id: 64c0ccc19832a455bc2a2069
paper_authors.avatarUrl: /avatars/9d4fdf13cd62c6c258b5f50d67c8a1b1.svg
paper_authors.fullname: IZZEDDIN GUR
paper_authors.type: user
paper_authors.user: izzeddingur
paper_authors._id: 6577f1e4ee33d547aebf89a1
paper_authors.name: Jasper Snoek
paper_authors._id: 6577f1e4ee33d547aebf89a2
paper_authors.name: Jeffrey Pennington
paper_authors._id: 6577f1e4ee33d547aebf89a3
paper_authors.name: Jiri Hron
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:26:27.286Z
paper_authors._id: 62af5f9e24488e6adfa86321
paper_authors.avatarUrl: /avatars/5990e5f656c75530e6e8f01f644f7c8d.svg
paper_authors.fullname: Jiri Hron
paper_authors.type: user
paper_authors.user: jirihron
paper_authors._id: 6577f1e4ee33d547aebf89a4
paper_authors.name: Kathleen Kenealy
paper_authors._id: 6577f1e4ee33d547aebf89a5
paper_authors.name: Kevin Swersky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:26:41.648Z
paper_authors._id: 630e6ef664f1f8d0c771b758
paper_authors.avatarUrl: /avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
paper_authors.fullname: Kevin Swersky
paper_authors.type: user
paper_authors.user: kswersky
paper_authors._id: 6577f1e4ee33d547aebf89a6
paper_authors.name: Kshiteej Mahajan
paper_authors._id: 6577f1e4ee33d547aebf89a7
paper_authors.name: Laura Culp
paper_authors._id: 6577f1e4ee33d547aebf89a8
paper_authors.name: Lechao Xiao
paper_authors._id: 6577f1e4ee33d547aebf89a9
paper_authors.name: Maxwell L. Bileschi
paper_authors._id: 6577f1e4ee33d547aebf89aa
paper_authors.name: Noah Constant
paper_authors._id: 6577f1e4ee33d547aebf89ab
paper_authors.name: Roman Novak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:27:20.964Z
paper_authors._id: 63e2769bf0740bec2bfff1a1
paper_authors.avatarUrl: /avatars/e2841a836319d43035bcddfbfe3344d0.svg
paper_authors.fullname: Roman Novak
paper_authors.type: user
paper_authors.user: kulpims
paper_authors._id: 6577f1e4ee33d547aebf89ac
paper_authors.name: Rosanne Liu
paper_authors._id: 6577f1e4ee33d547aebf89ad
paper_authors.name: Tris Warkentin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:27:44.017Z
paper_authors._id: 64d51be85e5f05485ca50751
paper_authors.avatarUrl: /avatars/5802fcc5a5f1e089df41232892f8f502.svg
paper_authors.fullname: Tris Warkentin
paper_authors.type: user
paper_authors.user: trisfromgoogle
paper_authors._id: 6577f1e4ee33d547aebf89ae
paper_authors.name: Yundi Qian
paper_authors._id: 6577f1e4ee33d547aebf89af
paper_authors.name: Ethan Dyer
paper_authors._id: 6577f1e4ee33d547aebf89b0
paper_authors.name: Behnam Neyshabur
paper_authors._id: 6577f1e4ee33d547aebf89b1
paper_authors.name: Jascha Sohl-Dickstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:28:09.903Z
paper_authors._id: 63d209ae8eaa4831006b3b0d
paper_authors.avatarUrl: /avatars/51cc0f6dfab2233bddd73bfcd0c8aee3.svg
paper_authors.fullname: Jascha Sohl-Dickstein
paper_authors.type: user
paper_authors.user: jaschasd
paper_authors._id: 6577f1e4ee33d547aebf89b2
paper_authors.name: Noah Fiedel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:28:18.023Z
paper_authors._id: 62f82ebd870a3f98bbf9e62a
paper_authors.avatarUrl: /avatars/3b03299e128abf2c8ba82db704855de7.svg
paper_authors.fullname: Noah Fiedel
paper_authors.type: user
paper_authors.user: nfiedel
paper_title: Beyond Human Data: Scaling Self-Training for Problem-Solving with
    Language Models
paper_summary: Fine-tuning language models~(LMs) on human-generated data remains a prevalent
  practice. However, the performance of such models is often limited by the
  quantity and diversity of high-quality human data. In this paper, we explore
  whether we can go beyond human data on tasks where we have access to scalar
  feedback, for example, on math problems where one can verify correctness. To do
  so, we investigate a simple self-training method based on
  expectation-maximization, which we call ReST^{EM}, where we (1) generate
  samples from the model and filter them using binary feedback, (2) fine-tune the
  model on these samples, and (3) repeat this process a few times. Testing on
  advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find
  that ReST^{EM} scales favorably with model size and significantly surpasses
  fine-tuning only on human data. Overall, our findings suggest self-training
  with feedback can substantially reduce dependence on human-generated data.
paper_ai_summary: Self-training with scalar feedback using expectation-maximization improves language model performance on tasks with verifiable answers, reducing reliance on human data.
paper_ai_keywords: self-training expectation-maximization scalar feedback MATH reasoning APPS coding benchmarks fine-tuning