Large Language Models of Code Fail at Completing Code with Potential
  Bugs
Large language models of code (Code-LLMs) have recently brought tremendous
advances to code completion, a fundamental feature of programming assistance
and code intelligence. However, most existing works ignore the possible
presence of bugs in the code context for generation, which are inevitable in
software development. Therefore, we introduce and study the buggy-code
completion problem, inspired by the realistic scenario of real-time code
suggestion where the code context contains potential bugs -- anti-patterns that
can become bugs in the completed program. To systematically study the task, we
introduce two datasets: one with synthetic bugs derived from semantics-altering
operator changes (buggy-HumanEval) and one with realistic bugs derived from
user submissions to coding problems (buggy-FixEval). We find that the presence
of potential bugs significantly degrades the generation performance of the
high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono
on test cases of buggy-HumanEval drop more than 50% given a single potential
bug in the context. Finally, we investigate several post-hoc methods for
mitigating the adverse effect of potential bugs and find that there remains a
large gap in post-mitigation performance.
647fdbcf28b737d7b9458861
Tuan Dinh
647fdbcf28b737d7b9458862
Jinman Zhao
647fdbcf28b737d7b9458863
Samson Tan
647fdbcf28b737d7b9458864
Renato Negrinho
647fdbcf28b737d7b9458865
Leonard Lausen
647fdbcf28b737d7b9458866
Sheng Zha
647fdbcf28b737d7b9458867
George Karypis
Large Language Models of Code Fail at Completing Code with Potential
  Bugs
Large language models of code (Code-LLMs) have recently brought tremendous
advances to code completion, a fundamental feature of programming assistance
and code intelligence. However, most existing works ignore the possible
presence of bugs in the code context for generation, which are inevitable in
software development. Therefore, we introduce and study the buggy-code
completion problem, inspired by the realistic scenario of real-time code
suggestion where the code context contains potential bugs -- anti-patterns that
can become bugs in the completed program. To systematically study the task, we
introduce two datasets: one with synthetic bugs derived from semantics-altering
operator changes (buggy-HumanEval) and one with realistic bugs derived from
user submissions to coding problems (buggy-FixEval). We find that the presence
of potential bugs significantly degrades the generation performance of the
high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono
on test cases of buggy-HumanEval drop more than 50% given a single potential
bug in the context. Finally, we investigate several post-hoc methods for
mitigating the adverse effect of potential bugs and find that there remains a
large gap in post-mitigation performance.
Code language models (Code-LLMs) show degraded performance in code completion tasks when the context includes potential bugs, and post-mitigation strategies do not fully address this issue.
Code-LLMs code completion buggy-code completion code context potential bugs anti-patterns buggy-HumanEval buggy-FixEval passing rates post-hoc methods