xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
With the release of the o1 model by OpenAI, reasoning models adopting slow
thinking strategies have gradually emerged. As the responses generated by such
models often include complex reasoning, intermediate steps, and
self-reflection, existing evaluation methods are often inadequate. They
struggle to determine whether the LLM output is truly equivalent to the
reference answer, and also have difficulty identifying and extracting the final
answer from long, complex responses. To address this issue, we propose xVerify,
an efficient answer verifier for reasoning model evaluations. xVerify
demonstrates strong capability in equivalence judgment, enabling it to
effectively determine whether the answers produced by reasoning models are
equivalent to reference answers across various types of objective questions. To
train and evaluate xVerify, we construct the VAR dataset by collecting
question-answer pairs generated by multiple LLMs across various datasets,
leveraging multiple reasoning models and challenging evaluation sets designed
specifically for reasoning model assessment. A multi-round annotation process
is employed to ensure label accuracy. Based on the VAR dataset, we train
multiple xVerify models of different scales. In evaluation experiments
conducted on both the test set and generalization set, all xVerify models
achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
validate the effectiveness and generalizability of xVerify.
67fdc1b41d1bc292f7b9358e
Ding Chen
claimed_verified
2025-04-15T07:54:22.449Z
64e18e9ec20c27fcc8df384e
/avatars/64ef866b9fa385efcefb34ea76b76802.svg
Ding Chen
user
Hush-cd
67fdc1b41d1bc292f7b9358f
Qingchen Yu
claimed_verified
2025-04-16T09:23:45.275Z
6455ff584095c967f9a847bb
https://cdn-avatars.huggingface.co/v1/production/uploads/6455ff584095c967f9a847bb/A5wjtWsudC73fLVmgASBr.jpeg
Qingchen Yu
user
Duguce
67fdc1b41d1bc292f7b93590
Pengyuan Wang
admin_assigned
2025-04-17T08:35:39.977Z
6443711ff99d6629a71e69b5
/avatars/5b4cd90484512376a24bfe5ffc39dbb4.svg
Peng Yuan wang
user
deflinhec
67fdc1b41d1bc292f7b93591
Wentao Zhang
67fdc1b41d1bc292f7b93592
Bo Tang
67fdc1b41d1bc292f7b93593
Feiyu Xiong
67fdc1b41d1bc292f7b93594
Xinchi Li
67fdc1b41d1bc292f7b93595
Minchuan Yang
67fdc1b41d1bc292f7b93596
Zhiyu Li
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
With the release of the o1 model by OpenAI, reasoning models adopting slow
thinking strategies have gradually emerged. As the responses generated by such
models often include complex reasoning, intermediate steps, and
self-reflection, existing evaluation methods are often inadequate. They
struggle to determine whether the LLM output is truly equivalent to the
reference answer, and also have difficulty identifying and extracting the final
answer from long, complex responses. To address this issue, we propose xVerify,
an efficient answer verifier for reasoning model evaluations. xVerify
demonstrates strong capability in equivalence judgment, enabling it to
effectively determine whether the answers produced by reasoning models are
equivalent to reference answers across various types of objective questions. To
train and evaluate xVerify, we construct the VAR dataset by collecting
question-answer pairs generated by multiple LLMs across various datasets,
leveraging multiple reasoning models and challenging evaluation sets designed
specifically for reasoning model assessment. A multi-round annotation process
is employed to ensure label accuracy. Based on the VAR dataset, we train
multiple xVerify models of different scales. In evaluation experiments
conducted on both the test set and generalization set, all xVerify models
achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
validate the effectiveness and generalizability of xVerify.
xVerify, an efficient answer verifier, demonstrates strong equivalence judgment capabilities in evaluating reasoning models across various objective questions, with F1 scores and accuracy surpassing other methods.
reasoning models equivalence judgment xVerify VAR dataset F1 scores