title: CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
summary: We present CRUXEval (Code Reasoning, Understanding, and eXecution
  Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
  function comes with an input-output pair, leading to two natural tasks: input
  prediction and output prediction. First, we propose a generic recipe for
  generating our execution benchmark which can be used to create future variation
  of the benchmark. Second, we evaluate twenty code models on our benchmark and
  discover that many recent high-scoring models on HumanEval do not show the same
  improvements on our benchmark. Third, we show that simple CoT and fine-tuning
  schemes can improve performance on our benchmark but remain far from solving
  it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75%
  and 81% on input and output prediction, respectively. In contrast, Code Llama
  34B achieves a pass@1 of 50% and 46% on input and output prediction,
  highlighting the gap between open and closed source models. As no model is
  close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
  simple programs as a lens into its code reasoning capabilities and areas for
  improvement.
paper_authors._id: 659cb947198200abaf9b6954
paper_authors.name: Alex Gu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-09T14:46:04.506Z
paper_authors._id: 6179abf8ec6ce4dc2e5f2376
paper_authors.avatarUrl: /avatars/d10c6a1b350146b36949a24220471295.svg
paper_authors.fullname: Alex Gu
paper_authors.type: user
paper_authors.user: minimario
paper_authors._id: 659cb947198200abaf9b6955
paper_authors.name: Baptiste Rozière
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:49:04.500Z
paper_authors._id: 62d063dac375d0c84255b9a1
paper_authors.avatarUrl: /avatars/de0fc34bad8c761210c0895ebfa4feba.svg
paper_authors.fullname: Baptiste Roziere
paper_authors.type: user
paper_authors.user: broz
paper_authors._id: 659cb947198200abaf9b6956
paper_authors.name: Hugh Leather
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-07T18:30:44.421Z
paper_authors._id: 6704285b8992df50025c6283
paper_authors.avatarUrl: /avatars/e10ac62c0c20de3487b1ad23f50d3457.svg
paper_authors.fullname: Hugh Leather
paper_authors.type: user
paper_authors.user: hleather
paper_authors._id: 659cb947198200abaf9b6957
paper_authors.name: Armando Solar-Lezama
paper_authors._id: 659cb947198200abaf9b6958
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:49:30.885Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 659cb947198200abaf9b6959
paper_authors.name: Sida I. Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:49:40.877Z
paper_authors._id: 62a9f09f097098dc397e28c2
paper_authors.avatarUrl: /avatars/f7a09d1aae490ca066eccc4f011edb8a.svg
paper_authors.fullname: Sida Wang
paper_authors.type: user
paper_authors.user: sidaw
paper_title: CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
paper_summary: We present CRUXEval (Code Reasoning, Understanding, and eXecution
  Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
  function comes with an input-output pair, leading to two natural tasks: input
  prediction and output prediction. First, we propose a generic recipe for
  generating our execution benchmark which can be used to create future variation
  of the benchmark. Second, we evaluate twenty code models on our benchmark and
  discover that many recent high-scoring models on HumanEval do not show the same
  improvements on our benchmark. Third, we show that simple CoT and fine-tuning
  schemes can improve performance on our benchmark but remain far from solving
  it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75%
  and 81% on input and output prediction, respectively. In contrast, Code Llama
  34B achieves a pass@1 of 50% and 46% on input and output prediction,
  highlighting the gap between open and closed source models. As no model is
  close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
  simple programs as a lens into its code reasoning capabilities and areas for
  improvement.
paper_ai_summary: CRUXEval, a benchmark of Python functions with input-output pairs, evaluates code models and reveals performance differences compared to HumanEval, with GPT-4 showing better results but still falling short.
paper_ai_keywords: Code Reasoning Understanding and eXecution Evaluation (CRUXEval) HumanEval chain of thought (CoT) fine-tuning pass@1 Code Llama