Predicting Code Coverage without Execution
Code coverage is a widely used metric for quantifying the extent to which
program elements, such as statements or branches, are executed during testing.
Calculating code coverage is resource-intensive, requiring code building and
execution with additional overhead for the instrumentation. Furthermore,
computing coverage of any snippet of code requires the whole program context.
Using Machine Learning to amortize this expensive process could lower the cost
of code coverage by requiring only the source code context, and the task of
code coverage prediction can be a novel benchmark for judging the ability of
models to understand code. We propose a novel benchmark task called Code
Coverage Prediction for Large Language Models (LLMs). We formalize this task to
evaluate the capability of LLMs in understanding code execution by determining
which lines of a method are executed by a given test case and inputs. We curate
and release a dataset we call COVERAGEEVAL by executing tests and code from the
HumanEval dataset and collecting code coverage information. We report the
performance of four state-of-the-art LLMs used for code-related tasks,
including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's
Claude, on the Code Coverage Prediction task. Finally, we argue that code
coverage as a metric and pre-training data source are valuable for overall LLM
performance on software engineering tasks.
64c080fc0cfdb492ce84d000
Michele Tufano
64c080fc0cfdb492ce84d001
Shubham Chandel
claimed_verified
2023-07-26T08:43:15.840Z
5f5dd444e56d546cd623316a
/avatars/6b0d8511acf44247f7810399f275b2f0.svg
Shubham Chandel
user
sksq96
64c080fc0cfdb492ce84d002
Anisha Agarwal
admin_assigned
2023-07-26T09:16:26.031Z
6446c31574ccd9582b6be959
/avatars/04f4933e00d261630714126933f465f5.svg
Anisha Agarwal
user
24anisha
64c080fc0cfdb492ce84d003
Neel Sundaresan
64c080fc0cfdb492ce84d004
Colin Clement
admin_assigned
2023-07-26T09:16:58.702Z
5e41d258fc778f6520cbd8db
/avatars/87b3ecfba4bdabc9bfb75fc415c001b3.svg
Colin Clement
user
colinclement
Predicting Code Coverage without Execution
Code coverage is a widely used metric for quantifying the extent to which
program elements, such as statements or branches, are executed during testing.
Calculating code coverage is resource-intensive, requiring code building and
execution with additional overhead for the instrumentation. Furthermore,
computing coverage of any snippet of code requires the whole program context.
Using Machine Learning to amortize this expensive process could lower the cost
of code coverage by requiring only the source code context, and the task of
code coverage prediction can be a novel benchmark for judging the ability of
models to understand code. We propose a novel benchmark task called Code
Coverage Prediction for Large Language Models (LLMs). We formalize this task to
evaluate the capability of LLMs in understanding code execution by determining
which lines of a method are executed by a given test case and inputs. We curate
and release a dataset we call COVERAGEEVAL by executing tests and code from the
HumanEval dataset and collecting code coverage information. We report the
performance of four state-of-the-art LLMs used for code-related tasks,
including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's
Claude, on the Code Coverage Prediction task. Finally, we argue that code
coverage as a metric and pre-training data source are valuable for overall LLM
performance on software engineering tasks.
Large Language Models (LLMs) Code Coverage Prediction Supervised learning Code navigation HumanFlexibility dataset COVERAGEEVAL Code coverage Test cases Inputs Code understanding Software engineering tasks