title: CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and
    Evolution
summary: Efficient and accurate evaluation is crucial for the continuous improvement
  of large language models (LLMs). Among various assessment methods, subjective
  evaluation has garnered significant attention due to its superior alignment
  with real-world usage scenarios and human preferences. However, human-based
  evaluations are costly and lack reproducibility, making precise automated
  evaluators (judgers) vital in this process. In this report, we introduce
  CompassJudger-1, the first open-source all-in-one judge LLM.
  CompassJudger-1 is a general-purpose LLM that demonstrates remarkable
  versatility. It is capable of: 1. Performing unitary scoring and two-model
  comparisons as a reward model; 2. Conducting evaluations according to specified
  formats; 3. Generating critiques; 4. Executing diverse tasks like a general
  LLM. To assess the evaluation capabilities of different judge models under a
  unified setting, we have also established JudgerBench, a new benchmark
  that encompasses various subjective evaluation tasks and covers a wide range of
  topics. CompassJudger-1 offers a comprehensive solution for various evaluation
  tasks while maintaining the flexibility to adapt to diverse requirements. Both
  CompassJudger and JudgerBench are released and available to the research
  community athttps://github.com/open-compass/CompassJudger. We believe that by
  open-sourcing these tools, we can foster collaboration and accelerate progress
  in LLM evaluation methodologies.
paper_authors._id: 671713d85d781c10ef4152b0
paper_authors.name: Maosong Cao
paper_authors._id: 671713d85d781c10ef4152b1
paper_authors.name: Alexander Lam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:37:21.537Z
paper_authors._id: 63fc0d040aab060792003a68
paper_authors.avatarUrl: /avatars/d573d4c4ec49513f3b157f7691becbdd.svg
paper_authors.fullname: Alexander Lam
paper_authors.type: user
paper_authors.user: acylam
paper_authors._id: 671713d85d781c10ef4152b2
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:54.994Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 671713d85d781c10ef4152b3
paper_authors.name: Hongwei Liu
paper_authors._id: 671713d85d781c10ef4152b4
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:56.898Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 671713d85d781c10ef4152b5
paper_authors.name: Kai Chen
paper_title: CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and
    Evolution
paper_summary: Efficient and accurate evaluation is crucial for the continuous improvement
  of large language models (LLMs). Among various assessment methods, subjective
  evaluation has garnered significant attention due to its superior alignment
  with real-world usage scenarios and human preferences. However, human-based
  evaluations are costly and lack reproducibility, making precise automated
  evaluators (judgers) vital in this process. In this report, we introduce
  CompassJudger-1, the first open-source all-in-one judge LLM.
  CompassJudger-1 is a general-purpose LLM that demonstrates remarkable
  versatility. It is capable of: 1. Performing unitary scoring and two-model
  comparisons as a reward model; 2. Conducting evaluations according to specified
  formats; 3. Generating critiques; 4. Executing diverse tasks like a general
  LLM. To assess the evaluation capabilities of different judge models under a
  unified setting, we have also established JudgerBench, a new benchmark
  that encompasses various subjective evaluation tasks and covers a wide range of
  topics. CompassJudger-1 offers a comprehensive solution for various evaluation
  tasks while maintaining the flexibility to adapt to diverse requirements. Both
  CompassJudger and JudgerBench are released and available to the research
  community athttps://github.com/open-compass/CompassJudger. We believe that by
  open-sourcing these tools, we can foster collaboration and accelerate progress
  in LLM evaluation methodologies.
paper_ai_summary: CompassJudger-1 is an open-source, all-in-one judge LLM that evaluates LLMs through unitary scoring, model comparisons, critiques, and various tasks, supported by the JudgerBench benchmark.
paper_ai_keywords: LLMs subjective evaluation automated evaluators reward model evaluations according to specified formats generating critiques JudgerBench benchmark evaluation tasks