title: Large Language Models of Code Fail at Completing Code with Potential
    Bugs
summary: Large language models of code (Code-LLMs) have recently brought tremendous
  advances to code completion, a fundamental feature of programming assistance
  and code intelligence. However, most existing works ignore the possible
  presence of bugs in the code context for generation, which are inevitable in
  software development. Therefore, we introduce and study the buggy-code
  completion problem, inspired by the realistic scenario of real-time code
  suggestion where the code context contains potential bugs -- anti-patterns that
  can become bugs in the completed program. To systematically study the task, we
  introduce two datasets: one with synthetic bugs derived from semantics-altering
  operator changes (buggy-HumanEval) and one with realistic bugs derived from
  user submissions to coding problems (buggy-FixEval). We find that the presence
  of potential bugs significantly degrades the generation performance of the
  high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono
  on test cases of buggy-HumanEval drop more than 50% given a single potential
  bug in the context. Finally, we investigate several post-hoc methods for
  mitigating the adverse effect of potential bugs and find that there remains a
  large gap in post-mitigation performance.
paper_authors._id: 647fdbcf28b737d7b9458861
paper_authors.name: Tuan Dinh
paper_authors._id: 647fdbcf28b737d7b9458862
paper_authors.name: Jinman Zhao
paper_authors._id: 647fdbcf28b737d7b9458863
paper_authors.name: Samson Tan
paper_authors._id: 647fdbcf28b737d7b9458864
paper_authors.name: Renato Negrinho
paper_authors._id: 647fdbcf28b737d7b9458865
paper_authors.name: Leonard Lausen
paper_authors._id: 647fdbcf28b737d7b9458866
paper_authors.name: Sheng Zha
paper_authors._id: 647fdbcf28b737d7b9458867
paper_authors.name: George Karypis
paper_title: Large Language Models of Code Fail at Completing Code with Potential
    Bugs
paper_summary: Large language models of code (Code-LLMs) have recently brought tremendous
  advances to code completion, a fundamental feature of programming assistance
  and code intelligence. However, most existing works ignore the possible
  presence of bugs in the code context for generation, which are inevitable in
  software development. Therefore, we introduce and study the buggy-code
  completion problem, inspired by the realistic scenario of real-time code
  suggestion where the code context contains potential bugs -- anti-patterns that
  can become bugs in the completed program. To systematically study the task, we
  introduce two datasets: one with synthetic bugs derived from semantics-altering
  operator changes (buggy-HumanEval) and one with realistic bugs derived from
  user submissions to coding problems (buggy-FixEval). We find that the presence
  of potential bugs significantly degrades the generation performance of the
  high-performing Code-LLMs. For instance, the passing rates of CodeGen-2B-mono
  on test cases of buggy-HumanEval drop more than 50% given a single potential
  bug in the context. Finally, we investigate several post-hoc methods for
  mitigating the adverse effect of potential bugs and find that there remains a
  large gap in post-mitigation performance.
paper_ai_keywords: Code-LLMs buggy-code completion anti-patterns buggy-HumanEval buggy-FixEval generation performance passing rates post-hoc methods