title: A Static Evaluation of Code Completion by Large Language Models
summary: Large language models trained on code have shown great potential to increase
  productivity of software developers. Several execution-based benchmarks have
  been proposed to evaluate functional correctness of model-generated code on
  simple programming problems. Nevertheless, it is expensive to perform the same
  evaluation on complex real-world projects considering the execution cost. On
  the contrary, static analysis tools such as linters, which can detect errors
  without running the program, haven't been well explored for evaluating code
  generation models. In this work, we propose a static evaluation framework to
  quantify static errors in Python code completions, by leveraging Abstract
  Syntax Trees. Compared with execution-based evaluation, our method is not only
  more efficient, but also applicable to code in the wild. For experiments, we
  collect code context from open source repos to generate one million function
  bodies using public models. Our static analysis reveals that Undefined Name and
  Unused Variable are the most common errors among others made by language
  models. Through extensive studies, we also show the impact of sampling
  temperature, model size, and context on static errors in code completions.
paper_authors._id: 647fdac2095af0bf116dbe90
paper_authors.name: Hantian Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:08:25.714Z
paper_authors._id: 637556dafd59a439194d819d
paper_authors.avatarUrl: /avatars/a6241a3adf3bd3b1c3a6ba63933a4117.svg
paper_authors.fullname: Hantian Ding
paper_authors.type: user
paper_authors.user: dhantian
paper_authors._id: 647fdac2095af0bf116dbe91
paper_authors.name: Varun Kumar
paper_authors._id: 647fdac2095af0bf116dbe92
paper_authors.name: Yuchen Tian
paper_authors._id: 647fdac2095af0bf116dbe93
paper_authors.name: Zijian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T14:13:25.831Z
paper_authors._id: 6465183934acce85aea2c3b3
paper_authors.avatarUrl: /avatars/275a0811381afc980d339ba2aa1294c7.svg
paper_authors.fullname: Zijian Wang
paper_authors.type: user
paper_authors.user: zijwang
paper_authors._id: 647fdac2095af0bf116dbe94
paper_authors.name: Rob Kwiatkowski
paper_authors._id: 647fdac2095af0bf116dbe95
paper_authors.name: Xiaopeng Li
paper_authors._id: 647fdac2095af0bf116dbe96
paper_authors.name: Murali Krishna Ramanathan
paper_authors._id: 647fdac2095af0bf116dbe97
paper_authors.name: Baishakhi Ray
paper_authors._id: 647fdac2095af0bf116dbe98
paper_authors.name: Parminder Bhatia
paper_authors._id: 647fdac2095af0bf116dbe99
paper_authors.name: Sudipta Sengupta
paper_authors._id: 647fdac2095af0bf116dbe9a
paper_authors.name: Dan Roth
paper_authors._id: 647fdac2095af0bf116dbe9b
paper_authors.name: Bing Xiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T14:14:53.849Z
paper_authors._id: 63e85894e02ee67e8e5a15dc
paper_authors.avatarUrl: /avatars/81d14a5b797d3e9e5c445dd0fd5680c1.svg
paper_authors.fullname: Bing Xiang
paper_authors.type: user
paper_authors.user: BingBingXiang
paper_title: A Static Evaluation of Code Completion by Large Language Models
paper_summary: Large language models trained on code have shown great potential to increase
  productivity of software developers. Several execution-based benchmarks have
  been proposed to evaluate functional correctness of model-generated code on
  simple programming problems. Nevertheless, it is expensive to perform the same
  evaluation on complex real-world projects considering the execution cost. On
  the contrary, static analysis tools such as linters, which can detect errors
  without running the program, haven't been well explored for evaluating code
  generation models. In this work, we propose a static evaluation framework to
  quantify static errors in Python code completions, by leveraging Abstract
  Syntax Trees. Compared with execution-based evaluation, our method is not only
  more efficient, but also applicable to code in the wild. For experiments, we
  collect code context from open source repos to generate one million function
  bodies using public models. Our static analysis reveals that Undefined Name and
  Unused Variable are the most common errors among others made by language
  models. Through extensive studies, we also show the impact of sampling
  temperature, model size, and context on static errors in code completions.
paper_ai_keywords: Abstract Syntax Trees (ASTs) static evaluation framework Undefined Name Unused Variable sampling temperature model size context static errors