title: Dedicated Feedback and Edit Models Empower Inference-Time Scaling for
    Open-Ended General-Domain Tasks
summary: Inference-Time Scaling has been critical to the success of recent models such
  as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for
  inference-time scaling require tasks to have answers that can be verified,
  limiting their application to domains such as math, coding and logical
  reasoning. We take inspiration from how humans make first attempts, ask for
  detailed feedback from others and make improvements based on such feedback
  across a wide spectrum of open-ended endeavors. To this end, we collect data
  for and train dedicated Feedback and Edit Models that are capable of performing
  inference-time scaling for open-ended general-domain tasks. In our setup, one
  model generates an initial response, which are given feedback by a second
  model, that are then used by a third model to edit the response. We show that
  performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo
  can be boosted by scaling the number of initial response drafts, effective
  feedback and edited responses. When scaled optimally, our setup based on 70B
  models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7
  as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and
  DeepSeek R1 with 92.3.
paper_authors._id: 67ca637e4cb4283da8ae2979
paper_authors.name: Zhilin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:03:28.183Z
paper_authors._id: 635865c8aff68f72ac039a4e
paper_authors.avatarUrl: /avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
paper_authors.fullname: Zhilin Wang
paper_authors.type: user
paper_authors.user: zhilinw
paper_authors._id: 67ca637e4cb4283da8ae297a
paper_authors.name: Jiaqi Zeng
paper_authors._id: 67ca637e4cb4283da8ae297b
paper_authors.name: Olivier Delalleau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:51:28.127Z
paper_authors._id: 6556379e10428134ff235afd
paper_authors.avatarUrl: /avatars/ec569729870d7392e806e59a02f37d0c.svg
paper_authors.fullname: Olivier Delalleau
paper_authors.type: user
paper_authors.user: odelalleau
paper_authors._id: 67ca637e4cb4283da8ae297c
paper_authors.name: Daniel Egert
paper_authors._id: 67ca637e4cb4283da8ae297d
paper_authors.name: Ellie Evans
paper_authors._id: 67ca637e4cb4283da8ae297e
paper_authors.name: Hoo-Chang Shin
paper_authors._id: 67ca637e4cb4283da8ae297f
paper_authors.name: Felipe Soares
paper_authors._id: 67ca637e4cb4283da8ae2980
paper_authors.name: Yi Dong
paper_authors._id: 67ca637e4cb4283da8ae2981
paper_authors.name: Oleksii Kuchaiev
paper_title: Dedicated Feedback and Edit Models Empower Inference-Time Scaling for
    Open-Ended General-Domain Tasks
paper_summary: Inference-Time Scaling has been critical to the success of recent models such
  as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for
  inference-time scaling require tasks to have answers that can be verified,
  limiting their application to domains such as math, coding and logical
  reasoning. We take inspiration from how humans make first attempts, ask for
  detailed feedback from others and make improvements based on such feedback
  across a wide spectrum of open-ended endeavors. To this end, we collect data
  for and train dedicated Feedback and Edit Models that are capable of performing
  inference-time scaling for open-ended general-domain tasks. In our setup, one
  model generates an initial response, which are given feedback by a second
  model, that are then used by a third model to edit the response. We show that
  performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo
  can be boosted by scaling the number of initial response drafts, effective
  feedback and edited responses. When scaled optimally, our setup based on 70B
  models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7
  as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and
  DeepSeek R1 with 92.3.
paper_ai_summary: Feedback and Edit Models trained for inference-time scaling improve performance on open-ended tasks, surpassing other models on a competitive benchmark.
paper_ai_keywords: Inference-Time Scaling Feedback and Edit Models Arena Hard Chatbot Arena Elo Llama 3 parameter-efficient fine-tuning