Study finds AI assistants help developers produce code likely to be buggy
Surprising. If only there were a way that we could have foreseen that an AI trained to write code in part by looking at people who, self-admittedly, don’t know how to write code, and people who write code for others with minimal context (Stack Overflow), would produce buggy code. It is a case of GIGO.Most developers do not learn much from Stack Overflow. Why do we expect AI to fare better? In my experience, one in ten (optimistically) Stack Overflow answers give anything more than a code snippet with enough information to get the asker through their immediate issue. It can be beneficial if you have the necessary understanding already and only want the code snippet, but this is no way for humans or machines to learn.Also, having an “AI Assistant” must lower programmers’ guards against buggy code. After all, it is an assistant - it must assist you, right? Subordinating humans to machines will not work in this domain until there is better training data and the machines can be taught the reason they are writing specific code. Until then, I have low hopes for AI-generated code.Even if AI could generate correct, bug-free code the majority (say 99.9% of the time), I expect finding and correcting bugs will be difficult for humans. For example, how many bugs are found and corrected by the author of code during development, versus how many in peer review? I’m reminded of a saying akin to “ask someone to review 5,000 lines of code: no bugs. Ask someone to review 5 lines of code: 5 bugs”. We are poor critical reviewers, and AI cannot fix that. AI assistants probably worsen reviews, because reviewers will expect high-quality code from their AI assistants.