title: How to Get Your LLM to Generate Challenging Problems for Evaluation
summary: The pace of evolution of Large Language Models (LLMs) necessitates new
  approaches for rigorous and comprehensive evaluation. Traditional human
  annotation is increasingly impracticable due to the complexities and costs
  involved in generating high-quality, challenging problems. In this work, we
  introduce CHASE, a unified framework to synthetically generate challenging
  problems using LLMs without human involvement. For a given task, our approach
  builds a hard problem in a bottom-up manner from simpler components. Moreover,
  our framework decomposes the generation process into independently verifiable
  sub-tasks, thereby ensuring a high level of quality and correctness. We
  implement CHASE to create evaluation benchmarks across three diverse domains:
  (1) document-based question answering, (2) repository-level code completion,
  and (3) math reasoning. The performance of state-of-the-art LLMs on these
  synthetic benchmarks lies in the range of 40-60% accuracy, thereby
  demonstrating the effectiveness of our framework at generating challenging
  problems. We publicly release our benchmarks and code.
paper_authors._id: 67b886298512a3eca0668ba6
paper_authors.name: Arkil Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:20:45.806Z
paper_authors._id: 631a523c04f8ed65eff16fb4
paper_authors.avatarUrl: /avatars/2b284403c88f140d7bef283f729f7a3e.svg
paper_authors.fullname: Arkil Patel
paper_authors.type: user
paper_authors.user: arkilpatel
paper_authors._id: 67b886298512a3eca0668ba7
paper_authors.name: Siva Reddy
paper_authors._id: 67b886298512a3eca0668ba8
paper_authors.name: Dzmitry Bahdanau
paper_title: How to Get Your LLM to Generate Challenging Problems for Evaluation
paper_summary: The pace of evolution of Large Language Models (LLMs) necessitates new
  approaches for rigorous and comprehensive evaluation. Traditional human
  annotation is increasingly impracticable due to the complexities and costs
  involved in generating high-quality, challenging problems. In this work, we
  introduce CHASE, a unified framework to synthetically generate challenging
  problems using LLMs without human involvement. For a given task, our approach
  builds a hard problem in a bottom-up manner from simpler components. Moreover,
  our framework decomposes the generation process into independently verifiable
  sub-tasks, thereby ensuring a high level of quality and correctness. We
  implement CHASE to create evaluation benchmarks across three diverse domains:
  (1) document-based question answering, (2) repository-level code completion,
  and (3) math reasoning. The performance of state-of-the-art LLMs on these
  synthetic benchmarks lies in the range of 40-60% accuracy, thereby
  demonstrating the effectiveness of our framework at generating challenging
  problems. We publicly release our benchmarks and code.
paper_ai_keywords: Large Language Models (LLMs) CHASE document-based question answering repository-level code completion math reasoning synthetic benchmarks high-level quality correctness