title: xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
summary: With the release of the o1 model by OpenAI, reasoning models adopting slow
  thinking strategies have gradually emerged. As the responses generated by such
  models often include complex reasoning, intermediate steps, and
  self-reflection, existing evaluation methods are often inadequate. They
  struggle to determine whether the LLM output is truly equivalent to the
  reference answer, and also have difficulty identifying and extracting the final
  answer from long, complex responses. To address this issue, we propose xVerify,
  an efficient answer verifier for reasoning model evaluations. xVerify
  demonstrates strong capability in equivalence judgment, enabling it to
  effectively determine whether the answers produced by reasoning models are
  equivalent to reference answers across various types of objective questions. To
  train and evaluate xVerify, we construct the VAR dataset by collecting
  question-answer pairs generated by multiple LLMs across various datasets,
  leveraging multiple reasoning models and challenging evaluation sets designed
  specifically for reasoning model assessment. A multi-round annotation process
  is employed to ensure label accuracy. Based on the VAR dataset, we train
  multiple xVerify models of different scales. In evaluation experiments
  conducted on both the test set and generalization set, all xVerify models
  achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
  variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
  while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
  validate the effectiveness and generalizability of xVerify.
paper_authors._id: 67fdc1b41d1bc292f7b9358e
paper_authors.name: Ding Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:22.449Z
paper_authors._id: 64e18e9ec20c27fcc8df384e
paper_authors.avatarUrl: /avatars/64ef866b9fa385efcefb34ea76b76802.svg
paper_authors.fullname: Ding Chen
paper_authors.type: user
paper_authors.user: Hush-cd
paper_authors._id: 67fdc1b41d1bc292f7b9358f
paper_authors.name: Qingchen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:45.275Z
paper_authors._id: 6455ff584095c967f9a847bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455ff584095c967f9a847bb/A5wjtWsudC73fLVmgASBr.jpeg
paper_authors.fullname: Qingchen Yu
paper_authors.type: user
paper_authors.user: Duguce
paper_authors._id: 67fdc1b41d1bc292f7b93590
paper_authors.name: Pengyuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:35:39.977Z
paper_authors._id: 6443711ff99d6629a71e69b5
paper_authors.avatarUrl: /avatars/5b4cd90484512376a24bfe5ffc39dbb4.svg
paper_authors.fullname: Peng Yuan wang
paper_authors.type: user
paper_authors.user: deflinhec
paper_authors._id: 67fdc1b41d1bc292f7b93591
paper_authors.name: Wentao Zhang
paper_authors._id: 67fdc1b41d1bc292f7b93592
paper_authors.name: Bo Tang
paper_authors._id: 67fdc1b41d1bc292f7b93593
paper_authors.name: Feiyu Xiong
paper_authors._id: 67fdc1b41d1bc292f7b93594
paper_authors.name: Xinchi Li
paper_authors._id: 67fdc1b41d1bc292f7b93595
paper_authors.name: Minchuan Yang
paper_authors._id: 67fdc1b41d1bc292f7b93596
paper_authors.name: Zhiyu Li
paper_title: xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
paper_summary: With the release of the o1 model by OpenAI, reasoning models adopting slow
  thinking strategies have gradually emerged. As the responses generated by such
  models often include complex reasoning, intermediate steps, and
  self-reflection, existing evaluation methods are often inadequate. They
  struggle to determine whether the LLM output is truly equivalent to the
  reference answer, and also have difficulty identifying and extracting the final
  answer from long, complex responses. To address this issue, we propose xVerify,
  an efficient answer verifier for reasoning model evaluations. xVerify
  demonstrates strong capability in equivalence judgment, enabling it to
  effectively determine whether the answers produced by reasoning models are
  equivalent to reference answers across various types of objective questions. To
  train and evaluate xVerify, we construct the VAR dataset by collecting
  question-answer pairs generated by multiple LLMs across various datasets,
  leveraging multiple reasoning models and challenging evaluation sets designed
  specifically for reasoning model assessment. A multi-round annotation process
  is employed to ensure label accuracy. Based on the VAR dataset, we train
  multiple xVerify models of different scales. In evaluation experiments
  conducted on both the test set and generalization set, all xVerify models
  achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
  variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
  while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
  validate the effectiveness and generalizability of xVerify.
paper_ai_summary: xVerify, an efficient answer verifier, demonstrates strong equivalence judgment capabilities in evaluating reasoning models across various objective questions, with F1 scores and accuracy surpassing other methods.
paper_ai_keywords: reasoning models equivalence judgment xVerify VAR dataset F1 scores