A Static Evaluation of Code Completion by Large Language Models
Large language models trained on code have shown great potential to increase
productivity of software developers. Several execution-based benchmarks have
been proposed to evaluate functional correctness of model-generated code on
simple programming problems. Nevertheless, it is expensive to perform the same
evaluation on complex real-world projects considering the execution cost. On
the contrary, static analysis tools such as linters, which can detect errors
without running the program, haven't been well explored for evaluating code
generation models. In this work, we propose a static evaluation framework to
quantify static errors in Python code completions, by leveraging Abstract
Syntax Trees. Compared with execution-based evaluation, our method is not only
more efficient, but also applicable to code in the wild. For experiments, we
collect code context from open source repos to generate one million function
bodies using public models. Our static analysis reveals that Undefined Name and
Unused Variable are the most common errors among others made by language
models. Through extensive studies, we also show the impact of sampling
temperature, model size, and context on static errors in code completions.
647fdac2095af0bf116dbe90
Hantian Ding
claimed_verified
2023-06-08T16:08:25.714Z
637556dafd59a439194d819d
/avatars/a6241a3adf3bd3b1c3a6ba63933a4117.svg
Hantian Ding
user
dhantian
647fdac2095af0bf116dbe91
Varun Kumar
647fdac2095af0bf116dbe92
Yuchen Tian
647fdac2095af0bf116dbe93
Zijian Wang
admin_assigned
2023-06-12T14:13:25.831Z
6465183934acce85aea2c3b3
/avatars/275a0811381afc980d339ba2aa1294c7.svg
Zijian Wang
user
zijwang
647fdac2095af0bf116dbe94
Rob Kwiatkowski
647fdac2095af0bf116dbe95
Xiaopeng Li
647fdac2095af0bf116dbe96
Murali Krishna Ramanathan
647fdac2095af0bf116dbe97
Baishakhi Ray
647fdac2095af0bf116dbe98
Parminder Bhatia
647fdac2095af0bf116dbe99
Sudipta Sengupta
647fdac2095af0bf116dbe9a
Dan Roth
647fdac2095af0bf116dbe9b
Bing Xiang
admin_assigned
2023-06-12T14:14:53.849Z
63e85894e02ee67e8e5a15dc
/avatars/81d14a5b797d3e9e5c445dd0fd5680c1.svg
Bing Xiang
user
BingBingXiang
A Static Evaluation of Code Completion by Large Language Models
Large language models trained on code have shown great potential to increase
productivity of software developers. Several execution-based benchmarks have
been proposed to evaluate functional correctness of model-generated code on
simple programming problems. Nevertheless, it is expensive to perform the same
evaluation on complex real-world projects considering the execution cost. On
the contrary, static analysis tools such as linters, which can detect errors
without running the program, haven't been well explored for evaluating code
generation models. In this work, we propose a static evaluation framework to
quantify static errors in Python code completions, by leveraging Abstract
Syntax Trees. Compared with execution-based evaluation, our method is not only
more efficient, but also applicable to code in the wild. For experiments, we
collect code context from open source repos to generate one million function
bodies using public models. Our static analysis reveals that Undefined Name and
Unused Variable are the most common errors among others made by language
models. Through extensive studies, we also show the impact of sampling
temperature, model size, and context on static errors in code completions.
A static evaluation framework using Abstract Syntax Trees quantifies static errors in Python code completions, offering a more efficient alternative to execution-based evaluation for large models.
Abstract Syntax Trees static analysis Undefined Name Unused Variable sampling temperature model size context