title: Predicting Code Coverage without Execution
summary: Code coverage is a widely used metric for quantifying the extent to which
  program elements, such as statements or branches, are executed during testing.
  Calculating code coverage is resource-intensive, requiring code building and
  execution with additional overhead for the instrumentation. Furthermore,
  computing coverage of any snippet of code requires the whole program context.
  Using Machine Learning to amortize this expensive process could lower the cost
  of code coverage by requiring only the source code context, and the task of
  code coverage prediction can be a novel benchmark for judging the ability of
  models to understand code. We propose a novel benchmark task called Code
  Coverage Prediction for Large Language Models (LLMs). We formalize this task to
  evaluate the capability of LLMs in understanding code execution by determining
  which lines of a method are executed by a given test case and inputs. We curate
  and release a dataset we call COVERAGEEVAL by executing tests and code from the
  HumanEval dataset and collecting code coverage information. We report the
  performance of four state-of-the-art LLMs used for code-related tasks,
  including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's
  Claude, on the Code Coverage Prediction task. Finally, we argue that code
  coverage as a metric and pre-training data source are valuable for overall LLM
  performance on software engineering tasks.
paper_authors._id: 64c080fc0cfdb492ce84d000
paper_authors.name: Michele Tufano
paper_authors._id: 64c080fc0cfdb492ce84d001
paper_authors.name: Shubham Chandel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T08:43:15.840Z
paper_authors._id: 5f5dd444e56d546cd623316a
paper_authors.avatarUrl: /avatars/6b0d8511acf44247f7810399f275b2f0.svg
paper_authors.fullname: Shubham Chandel
paper_authors.type: user
paper_authors.user: sksq96
paper_authors._id: 64c080fc0cfdb492ce84d002
paper_authors.name: Anisha Agarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T09:16:26.031Z
paper_authors._id: 6446c31574ccd9582b6be959
paper_authors.avatarUrl: /avatars/04f4933e00d261630714126933f465f5.svg
paper_authors.fullname: Anisha Agarwal
paper_authors.type: user
paper_authors.user: 24anisha
paper_authors._id: 64c080fc0cfdb492ce84d003
paper_authors.name: Neel Sundaresan
paper_authors._id: 64c080fc0cfdb492ce84d004
paper_authors.name: Colin Clement
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T09:16:58.702Z
paper_authors._id: 5e41d258fc778f6520cbd8db
paper_authors.avatarUrl: /avatars/87b3ecfba4bdabc9bfb75fc415c001b3.svg
paper_authors.fullname: Colin Clement
paper_authors.type: user
paper_authors.user: colinclement
paper_title: Predicting Code Coverage without Execution
paper_summary: Code coverage is a widely used metric for quantifying the extent to which
  program elements, such as statements or branches, are executed during testing.
  Calculating code coverage is resource-intensive, requiring code building and
  execution with additional overhead for the instrumentation. Furthermore,
  computing coverage of any snippet of code requires the whole program context.
  Using Machine Learning to amortize this expensive process could lower the cost
  of code coverage by requiring only the source code context, and the task of
  code coverage prediction can be a novel benchmark for judging the ability of
  models to understand code. We propose a novel benchmark task called Code
  Coverage Prediction for Large Language Models (LLMs). We formalize this task to
  evaluate the capability of LLMs in understanding code execution by determining
  which lines of a method are executed by a given test case and inputs. We curate
  and release a dataset we call COVERAGEEVAL by executing tests and code from the
  HumanEval dataset and collecting code coverage information. We report the
  performance of four state-of-the-art LLMs used for code-related tasks,
  including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's
  Claude, on the Code Coverage Prediction task. Finally, we argue that code
  coverage as a metric and pre-training data source are valuable for overall LLM
  performance on software engineering tasks.
paper_ai_keywords: Large Language Models (LLMs) Code Coverage Prediction Supervised learning Code navigation HumanFlexibility dataset COVERAGEEVAL Code coverage Test cases Inputs Code understanding Software engineering tasks