title: Benchmarking Large Language Models for Multi-Language Software
    Vulnerability Detection
summary: Recent advancements in generative AI have led to the widespread adoption of
  large language models (LLMs) in software engineering, addressing numerous
  long-standing challenges. However, a comprehensive study examining the
  capabilities of LLMs in software vulnerability detection (SVD), a crucial
  aspect of software security, is currently lacking. Existing research primarily
  focuses on evaluating LLMs using C/C++ datasets. It typically explores only one
  or two strategies among prompt engineering, instruction tuning, and sequence
  classification fine-tuning for open-source LLMs. Consequently, there is a
  significant knowledge gap regarding the effectiveness of diverse LLMs in
  detecting vulnerabilities across various programming languages. To address this
  knowledge gap, we present a comprehensive empirical study evaluating the
  performance of LLMs on the SVD task. We have compiled a comprehensive dataset
  comprising 8,260 vulnerable functions in Python, 7,505 in Java, and 28,983 in
  JavaScript. We assess five open-source LLMs using multiple approaches,
  including prompt engineering, instruction tuning, and sequence classification
  fine-tuning. These LLMs are benchmarked against five fine-tuned small language
  models and two open-source static application security testing tools.
  Furthermore, we explore two avenues to improve LLM performance on SVD: a) Data
  perspective: Retraining models using downsampled balanced datasets. b) Model
  perspective: Investigating ensemble learning methods that combine predictions
  from multiple LLMs. Our comprehensive experiments demonstrate that SVD remains
  a challenging task for LLMs. This study provides a thorough understanding of
  the role of LLMs in SVD and offers practical insights for future advancements
  in leveraging generative AI to enhance software security practices.
paper_authors._id: 67c92e738d5fe8c860571103
paper_authors.name: Ting Zhang
paper_authors._id: 67c92e738d5fe8c860571104
paper_authors.name: Chengran Yang
paper_authors._id: 67c92e738d5fe8c860571105
paper_authors.name: Yindu Su
paper_authors._id: 67c92e738d5fe8c860571106
paper_authors.name: Martin Weyssow
paper_authors._id: 67c92e738d5fe8c860571107
paper_authors.name: Hung Nguyen
paper_authors._id: 67c92e738d5fe8c860571108
paper_authors.name: Tan Bui
paper_authors._id: 67c92e738d5fe8c860571109
paper_authors.name: Hong Jin Kang
paper_authors._id: 67c92e738d5fe8c86057110a
paper_authors.name: Yikun Li
paper_authors._id: 67c92e738d5fe8c86057110b
paper_authors.name: Eng Lieh Ouh
paper_authors._id: 67c92e738d5fe8c86057110c
paper_authors.name: Lwin Khin Shar
paper_authors._id: 67c92e738d5fe8c86057110d
paper_authors.name: David Lo
paper_title: Benchmarking Large Language Models for Multi-Language Software
    Vulnerability Detection
paper_summary: Recent advancements in generative AI have led to the widespread adoption of
  large language models (LLMs) in software engineering, addressing numerous
  long-standing challenges. However, a comprehensive study examining the
  capabilities of LLMs in software vulnerability detection (SVD), a crucial
  aspect of software security, is currently lacking. Existing research primarily
  focuses on evaluating LLMs using C/C++ datasets. It typically explores only one
  or two strategies among prompt engineering, instruction tuning, and sequence
  classification fine-tuning for open-source LLMs. Consequently, there is a
  significant knowledge gap regarding the effectiveness of diverse LLMs in
  detecting vulnerabilities across various programming languages. To address this
  knowledge gap, we present a comprehensive empirical study evaluating the
  performance of LLMs on the SVD task. We have compiled a comprehensive dataset
  comprising 8,260 vulnerable functions in Python, 7,505 in Java, and 28,983 in
  JavaScript. We assess five open-source LLMs using multiple approaches,
  including prompt engineering, instruction tuning, and sequence classification
  fine-tuning. These LLMs are benchmarked against five fine-tuned small language
  models and two open-source static application security testing tools.
  Furthermore, we explore two avenues to improve LLM performance on SVD: a) Data
  perspective: Retraining models using downsampled balanced datasets. b) Model
  perspective: Investigating ensemble learning methods that combine predictions
  from multiple LLMs. Our comprehensive experiments demonstrate that SVD remains
  a challenging task for LLMs. This study provides a thorough understanding of
  the role of LLMs in SVD and offers practical insights for future advancements
  in leveraging generative AI to enhance software security practices.
paper_ai_summary: A study evaluates the performance of large language models in software vulnerability detection across multiple programming languages using various fine-tuning methods and compares them to smaller models and traditional tools.
paper_ai_keywords: large language models software vulnerability detection prompt engineering instruction tuning sequence classification fine-tuning ensemble learning