CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and
  Outcome Reward
Answer verification is crucial not only for evaluating large language models
(LLMs) by matching their unstructured outputs against standard answers, but
also serves as the reward model to guide LLM optimization. Most evaluation
frameworks rely on regularized matching or employ general LLMs for answer
verification, which demands extensive, repetitive customization for regex rules
or evaluation prompts. Two fundamental limitations persist in current
methodologies: 1) the absence of comprehensive benchmarks that systematically
evaluate verification capabilities across different LLMs; and 2) the nascent
stage of verifier development, where existing approaches lack both the
robustness to handle complex edge cases and the generalizability across
different domains. In this work, we develop CompassVerifier, an accurate and
robust lightweight verifier model for evaluation and outcome reward. It
demonstrates multi-domain competency spanning math, knowledge, and diverse
reasoning tasks, with the capability to process various answer types, including
multi-subproblems, formulas, and sequence answers, while effectively
identifying abnormal/invalid responses. We introduce VerifierBench benchmark
comprising model outputs collected from multiple data sources, augmented
through manual analysis of metaerror patterns to enhance CompassVerifier. We
anticipate that CompassVerifier and VerifierBench will facilitate answer
verification, evaluation protocols, and reinforcement learning research. Code
and dataset are available at https://github.com/open-compass/CompassVerifier.
6892f2468da45ffb0a2b2500
Shudong Liu
6892f2468da45ffb0a2b2501
Hongwei Liu
6892f2468da45ffb0a2b2502
Junnan Liu
6892f2468da45ffb0a2b2503
Linchen Xiao
6892f2468da45ffb0a2b2504
Songyang Gao
6892f2468da45ffb0a2b2505
Chengqi Lyu
6892f2468da45ffb0a2b2506
Yuzhe Gu
6892f2468da45ffb0a2b2507
Wenwei Zhang
6892f2468da45ffb0a2b2508
Derek F. Wong
6892f2468da45ffb0a2b2509
Songyang Zhang
6892f2468da45ffb0a2b250a
Kai Chen
CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and
  Outcome Reward
Answer verification is crucial not only for evaluating large language models
(LLMs) by matching their unstructured outputs against standard answers, but
also serves as the reward model to guide LLM optimization. Most evaluation
frameworks rely on regularized matching or employ general LLMs for answer
verification, which demands extensive, repetitive customization for regex rules
or evaluation prompts. Two fundamental limitations persist in current
methodologies: 1) the absence of comprehensive benchmarks that systematically
evaluate verification capabilities across different LLMs; and 2) the nascent
stage of verifier development, where existing approaches lack both the
robustness to handle complex edge cases and the generalizability across
different domains. In this work, we develop CompassVerifier, an accurate and
robust lightweight verifier model for evaluation and outcome reward. It
demonstrates multi-domain competency spanning math, knowledge, and diverse
reasoning tasks, with the capability to process various answer types, including
multi-subproblems, formulas, and sequence answers, while effectively
identifying abnormal/invalid responses. We introduce VerifierBench benchmark
comprising model outputs collected from multiple data sources, augmented
through manual analysis of metaerror patterns to enhance CompassVerifier. We
anticipate that CompassVerifier and VerifierBench will facilitate answer
verification, evaluation protocols, and reinforcement learning research. Code
and dataset are available at https://github.com/open-compass/CompassVerifier.
CompassVerifier is a lightweight, robust model for verifying LLM outputs across various domains, supported by VerifierBench, a comprehensive benchmark dataset.
LLMs answer verification reward model evaluation frameworks regex rules evaluation prompts benchmarks verifier development multi-domain competency math knowledge reasoning tasks multi-subproblems formulas sequence answers abnormal/invalid responses VerifierBench