CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
We present CRUXEval (Code Reasoning, Understanding, and eXecution
Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
function comes with an input-output pair, leading to two natural tasks: input
prediction and output prediction. First, we propose a generic recipe for
generating our execution benchmark which can be used to create future variation
of the benchmark. Second, we evaluate twenty code models on our benchmark and
discover that many recent high-scoring models on HumanEval do not show the same
improvements on our benchmark. Third, we show that simple CoT and fine-tuning
schemes can improve performance on our benchmark but remain far from solving
it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75%
and 81% on input and output prediction, respectively. In contrast, Code Llama
34B achieves a pass@1 of 50% and 46% on input and output prediction,
highlighting the gap between open and closed source models. As no model is
close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
simple programs as a lens into its code reasoning capabilities and areas for
improvement.
659cb947198200abaf9b6954
Alex Gu
extracted_confirmed
2024-01-09T14:46:04.506Z
6179abf8ec6ce4dc2e5f2376
/avatars/d10c6a1b350146b36949a24220471295.svg
Alex Gu
user
minimario
659cb947198200abaf9b6955
Baptiste Rozière
admin_assigned
2024-01-09T08:49:04.500Z
62d063dac375d0c84255b9a1
/avatars/de0fc34bad8c761210c0895ebfa4feba.svg
Baptiste Roziere
user
broz
659cb947198200abaf9b6956
Hugh Leather
extracted_pending
2024-10-07T18:30:44.421Z
6704285b8992df50025c6283
/avatars/e10ac62c0c20de3487b1ad23f50d3457.svg
Hugh Leather
user
hleather
659cb947198200abaf9b6957
Armando Solar-Lezama
659cb947198200abaf9b6958
Gabriel Synnaeve
admin_assigned
2024-01-09T08:49:30.885Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
659cb947198200abaf9b6959
Sida I. Wang
admin_assigned
2024-01-09T08:49:40.877Z
62a9f09f097098dc397e28c2
/avatars/f7a09d1aae490ca066eccc4f011edb8a.svg
Sida Wang
user
sidaw
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
We present CRUXEval (Code Reasoning, Understanding, and eXecution
Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
function comes with an input-output pair, leading to two natural tasks: input
prediction and output prediction. First, we propose a generic recipe for
generating our execution benchmark which can be used to create future variation
of the benchmark. Second, we evaluate twenty code models on our benchmark and
discover that many recent high-scoring models on HumanEval do not show the same
improvements on our benchmark. Third, we show that simple CoT and fine-tuning
schemes can improve performance on our benchmark but remain far from solving
it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75%
and 81% on input and output prediction, respectively. In contrast, Code Llama
34B achieves a pass@1 of 50% and 46% on input and output prediction,
highlighting the gap between open and closed source models. As no model is
close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
simple programs as a lens into its code reasoning capabilities and areas for
improvement.
CRUXEval, a benchmark of Python functions with input-output pairs, evaluates code models and reveals performance differences compared to HumanEval, with GPT-4 showing better results but still falling short.
Code Reasoning Understanding and eXecution Evaluation (CRUXEval) HumanEval chain of thought (CoT) fine-tuning pass@1 Code Llama