title: Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming
summary: The integration of Large Language Models (LLMs) into Development Environments
  (IDEs) has become a focal point in modern software development. LLMs such as
  OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment
  developer productivity by serving as intelligent, chat-driven programming
  assistants. However, utilizing LLMs out of the box is unlikely to be optimal
  for any given scenario. Rather, each system requires the LLM to be honed to its
  set of heuristics to ensure the best performance. In this paper, we introduce
  the Copilot evaluation harness: a set of data and tools for evaluating
  LLM-guided IDE interactions, covering various programming scenarios and
  languages. We propose our metrics as a more robust and information-dense
  evaluation than previous state of the art evaluation systems. We design and
  compute both static and execution based success metrics for scenarios
  encompassing a wide range of developer tasks, including code generation from
  natural language (generate), documentation generation from code (doc), test
  case generation (test), bug-fixing (fix), and workspace understanding and query
  resolution (workspace). These success metrics are designed to evaluate the
  performance of LLMs within a given IDE and its respective parameter space. Our
  learnings from evaluating three common LLMs using these metrics can inform the
  development and validation of future scenarios in LLM guided IDEs.
paper_authors._id: 65d80fcaab0790921b3141d6
paper_authors.name: Anisha Agarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:07:11.310Z
paper_authors._id: 6446c31574ccd9582b6be959
paper_authors.avatarUrl: /avatars/04f4933e00d261630714126933f465f5.svg
paper_authors.fullname: Anisha Agarwal
paper_authors.type: user
paper_authors.user: 24anisha
paper_authors._id: 65d80fcaab0790921b3141d7
paper_authors.name: Aaron Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:07:23.655Z
paper_authors._id: 6426440722bddcea3d27ee8a
paper_authors.avatarUrl: /avatars/daac33a79a6b35c36154274b6f109f8c.svg
paper_authors.fullname: Aaron_Chan
paper_authors.type: user
paper_authors.user: AaronChan
paper_authors._id: 65d80fcaab0790921b3141d8
paper_authors.name: Shubham Chandel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:07:42.487Z
paper_authors._id: 5f5dd444e56d546cd623316a
paper_authors.avatarUrl: /avatars/6b0d8511acf44247f7810399f275b2f0.svg
paper_authors.fullname: Shubham Chandel
paper_authors.type: user
paper_authors.user: sksq96
paper_authors._id: 65d80fcaab0790921b3141d9
paper_authors.name: Jinu Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:07:49.280Z
paper_authors._id: 644273eaa3b44a7571f930fd
paper_authors.avatarUrl: /avatars/0946be0d5e8d87224fe923a9f4833bb1.svg
paper_authors.fullname: Jinu Jang
paper_authors.type: user
paper_authors.user: ethanj-dev
paper_authors._id: 65d80fcaab0790921b3141da
paper_authors.name: Shaun Miller
paper_authors._id: 65d80fcaab0790921b3141db
paper_authors.name: Roshanak Zilouchian Moghaddam
paper_authors._id: 65d80fcaab0790921b3141dc
paper_authors.name: Yevhen Mohylevskyy
paper_authors._id: 65d80fcaab0790921b3141dd
paper_authors.name: Neel Sundaresan
paper_authors._id: 65d80fcaab0790921b3141de
paper_authors.name: Michele Tufano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:08:22.454Z
paper_authors._id: 65cee0e59aa49d5c96409bc7
paper_authors.avatarUrl: /avatars/69c8a0b28ab20c83fa30cbd9d427246d.svg
paper_authors.fullname: Michele Tufano
paper_authors.type: user
paper_authors.user: michele081tufano
paper_title: Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming
paper_summary: The integration of Large Language Models (LLMs) into Development Environments
  (IDEs) has become a focal point in modern software development. LLMs such as
  OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment
  developer productivity by serving as intelligent, chat-driven programming
  assistants. However, utilizing LLMs out of the box is unlikely to be optimal
  for any given scenario. Rather, each system requires the LLM to be honed to its
  set of heuristics to ensure the best performance. In this paper, we introduce
  the Copilot evaluation harness: a set of data and tools for evaluating
  LLM-guided IDE interactions, covering various programming scenarios and
  languages. We propose our metrics as a more robust and information-dense
  evaluation than previous state of the art evaluation systems. We design and
  compute both static and execution based success metrics for scenarios
  encompassing a wide range of developer tasks, including code generation from
  natural language (generate), documentation generation from code (doc), test
  case generation (test), bug-fixing (fix), and workspace understanding and query
  resolution (workspace). These success metrics are designed to evaluate the
  performance of LLMs within a given IDE and its respective parameter space. Our
  learnings from evaluating three common LLMs using these metrics can inform the
  development and validation of future scenarios in LLM guided IDEs.
paper_ai_keywords: Self Large Language Models (LLMs) OpenAI GPT-3.5/4 Code Llama intelligent, chat-driven programming assistants Copilot evaluation harness static success metrics execution based success metrics code generation from natural language documentation generation from code test case generation bug-fixing workspace understanding and query resolution parameter space