title: J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
summary: The progress of AI is bottlenecked by the quality of evaluation, and powerful
  LLM-as-a-Judge models have proved to be a core solution. Improved judgment
  ability is enabled by stronger chain-of-thought reasoning, motivating the need
  to find the best recipes for training such models to think. In this work we
  introduce J1, a reinforcement learning approach to training such models. Our
  method converts both verifiable and non-verifiable prompts to judgment tasks
  with verifiable rewards that incentivize thinking and mitigate judgment bias.
  In particular, our approach outperforms all other existing 8B or 70B models
  when trained at those sizes, including models distilled from DeepSeek-R1. J1
  also outperforms o1-mini, and even R1 on some benchmarks, despite training a
  smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
  Pointwise-J1 models, offline vs online training recipes, reward strategies,
  seed prompts, and variations in thought length and content. We find that our
  models make better judgments by learning to outline evaluation criteria,
  comparing against self-generated reference answers, and re-evaluating the
  correctness of model responses.
paper_authors._id: 6826a180cad9000ebc70f038
paper_authors.name: Chenxi Whitehouse
paper_authors._id: 6826a180cad9000ebc70f039
paper_authors.name: Tianlu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:42:23.084Z
paper_authors._id: 63f6a0e6b4c9a104f4b95d86
paper_authors.avatarUrl: /avatars/660c3e44bea976f3e0560bf95466b03c.svg
paper_authors.fullname: Tianlu Wang
paper_authors.type: user
paper_authors.user: Tianlu
paper_authors._id: 6826a180cad9000ebc70f03a
paper_authors.name: Ping Yu
paper_authors._id: 6826a180cad9000ebc70f03b
paper_authors.name: Xian Li
paper_authors._id: 6826a180cad9000ebc70f03c
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:42:55.728Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 6826a180cad9000ebc70f03d
paper_authors.name: Ilia Kulikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:43:03.749Z
paper_authors._id: 65d645913bc37244c55c39d5
paper_authors.avatarUrl: /avatars/508372876a654957650ee9429886d10c.svg
paper_authors.fullname: ilia kulikov
paper_authors.type: user
paper_authors.user: uralik
paper_authors._id: 6826a180cad9000ebc70f03e
paper_authors.name: Swarnadeep Saha
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-16T02:22:57.318Z
paper_authors._id: 64b75f4b037d6452a30f71aa
paper_authors.avatarUrl: /avatars/5a0322e7ecda05164e45526d605e3619.svg
paper_authors.fullname: Swarnadeep Saha
paper_authors.type: user
paper_authors.user: swarna92
paper_title: J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
paper_summary: The progress of AI is bottlenecked by the quality of evaluation, and powerful
  LLM-as-a-Judge models have proved to be a core solution. Improved judgment
  ability is enabled by stronger chain-of-thought reasoning, motivating the need
  to find the best recipes for training such models to think. In this work we
  introduce J1, a reinforcement learning approach to training such models. Our
  method converts both verifiable and non-verifiable prompts to judgment tasks
  with verifiable rewards that incentivize thinking and mitigate judgment bias.
  In particular, our approach outperforms all other existing 8B or 70B models
  when trained at those sizes, including models distilled from DeepSeek-R1. J1
  also outperforms o1-mini, and even R1 on some benchmarks, despite training a
  smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
  Pointwise-J1 models, offline vs online training recipes, reward strategies,
  seed prompts, and variations in thought length and content. We find that our
  models make better judgments by learning to outline evaluation criteria,
  comparing against self-generated reference answers, and re-evaluating the
  correctness of model responses.
paper_ai_summary: A reinforcement learning approach called J1 improves the judgment ability of LLM-as-a-Judge models through verifiable rewards and chain-of-thought reasoning.
paper_ai_keywords: reinforcement learning chain-of-thought reasoning judgment tasks reward strategies offline training online training Pairwise-J1 Pointwise-J1 evaluation criteria self-generated reference answers re-evaluation