title: HardTests: Synthesizing High-Quality Test Cases for LLM Coding
summary: Verifiers play a crucial role in large language model (LLM) reasoning, needed
  by post-training techniques such as reinforcement learning. However, reliable
  verifiers are hard to get for difficult coding problems, because a
  well-disguised wrong solution may only be detected by carefully human-written
  edge cases that are difficult to synthesize. To address this issue, we propose
  HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this
  pipeline, we curate a comprehensive competitive programming dataset HARDTESTS
  with 47k problems and synthetic high-quality tests. Compared with existing
  tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points
  higher and recall that is 17.5 percentage points higher when evaluating
  LLM-generated code. For harder problems, the improvement in precision can be as
  large as 40 points. HARDTESTS also proves to be more effective for model
  training, measured by downstream code generation performance. We will
  open-source our dataset and synthesis pipeline at
  https://leililab.github.io/HardTests/.
paper_authors._id: 683d2cee5bdbb3803e42bc8a
paper_authors.name: Zhongmou He
paper_authors._id: 683d2cee5bdbb3803e42bc8b
paper_authors.name: Yee Man Choi
paper_authors._id: 683d2cee5bdbb3803e42bc8c
paper_authors.name: Kexun Zhang
paper_authors._id: 683d2cee5bdbb3803e42bc8d
paper_authors.name: Jiabao Ji
paper_authors._id: 683d2cee5bdbb3803e42bc8e
paper_authors.name: Junting Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:40:34.926Z
paper_authors._id: 65a374a59acab1998092a9bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a374a59acab1998092a9bc/M3s_7bSf9G-6b9nLg7N3Z.jpeg
paper_authors.fullname: Antonio
paper_authors.type: user
paper_authors.user: JuntingZhou
paper_authors._id: 683d2cee5bdbb3803e42bc8f
paper_authors.name: Dejia Xu
paper_authors._id: 683d2cee5bdbb3803e42bc90
paper_authors.name: Ivan Bercovich
paper_authors._id: 683d2cee5bdbb3803e42bc91
paper_authors.name: Aidan Zhang
paper_authors._id: 683d2cee5bdbb3803e42bc92
paper_authors.name: Lei Li
paper_title: HardTests: Synthesizing High-Quality Test Cases for LLM Coding
paper_summary: Verifiers play a crucial role in large language model (LLM) reasoning, needed
  by post-training techniques such as reinforcement learning. However, reliable
  verifiers are hard to get for difficult coding problems, because a
  well-disguised wrong solution may only be detected by carefully human-written
  edge cases that are difficult to synthesize. To address this issue, we propose
  HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this
  pipeline, we curate a comprehensive competitive programming dataset HARDTESTS
  with 47k problems and synthetic high-quality tests. Compared with existing
  tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points
  higher and recall that is 17.5 percentage points higher when evaluating
  LLM-generated code. For harder problems, the improvement in precision can be as
  large as 40 points. HARDTESTS also proves to be more effective for model
  training, measured by downstream code generation performance. We will
  open-source our dataset and synthesis pipeline at
  https://leililab.github.io/HardTests/.
paper_ai_summary: HARDTESTGEN creates a large, high-quality competitive programming dataset to enhance the precision and recall of verifiers in evaluating LLM-generated code.
paper_ai_keywords: LLM reasoning reinforcement learning verifiers test synthesis LLMs competitive programming synthetic tests precision recall code generation performance