Automatic Detection of LLM-generated Code: A Case Study of Claude 3 Haiku
<|reference_start|>Automatic Detection of LLM-generated Code: A Case Study of Claude 3 Haiku: Using Large Language Models (LLMs) has gained popularity among software developers for generating source code. However, the use of LLM-generated code can introduce risks of adding suboptimal, defective, and vulnerable code. This makes it necessary to devise methods for the accurate detection of LLM-generated code. Toward this goal, we perform a case study of Claude 3 Haiku (or Claude 3 for brevity) on CodeSearchNet dataset. We divide our analyses into two parts: function-level and class-level. We extract 22 software metric features, such as Code Lines and Cyclomatic Complexity, for each level of granularity. We then analyze code snippets generated by Claude 3 and their human-authored counterparts using the extracted features to understand how unique the code generated by Claude 3 is. In the following step, we use the unique characteristics of Claude 3-generated code to build Machine Learning (ML) models and identify which features of the code snippets make them more detectable by ML models. Our results indicate that Claude 3 tends to generate longer functions, but shorter classes than humans, and this characteristic can be used to detect Claude 3-generated code with ML models with 82% and 66% accuracies for function-level and class-level snippets, respectively.<|reference_end|>
@article{rahman2024automatic,
    title={Automatic Detection of LLM-generated Code: A Case Study of Claude 3
  Haiku},
    author={Musfiqur Rahman, SayedHassan Khatoonabadi, Ahmad Abdellatif, Emad
  Shihab},
    journal={arXiv preprint arXiv:2409.01382},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01382},
    primaryClass={cs.SE cs.AI cs.LG}
}