title: Transformer-based Vulnerability Detection in Code at EditTime:
    Zero-shot, Few-shot, or Fine-tuning?
summary: Software vulnerabilities bear enterprises significant costs. Despite
  extensive efforts in research and development of software vulnerability
  detection methods, uncaught vulnerabilities continue to put software owners and
  users at risk. Many current vulnerability detection methods require that code
  snippets can compile and build before attempting detection. This,
  unfortunately, introduces a long latency between the time a vulnerability is
  injected to the time it is removed, which can substantially increases the cost
  of fixing a vulnerability. We recognize that the current advances in machine
  learning can be used to detect vulnerable code patterns on syntactically
  incomplete code snippets as the developer is writing the code at EditTime. In
  this paper we present a practical system that leverages deep learning on a
  large-scale data set of vulnerable code patterns to learn complex
  manifestations of more than 250 vulnerability types and detect vulnerable code
  patterns at EditTime. We discuss zero-shot, few-shot, and fine-tuning
  approaches on state of the art pre-trained Large Language Models (LLMs). We
  show that in comparison with state of the art vulnerability detection models
  our approach improves the state of the art by 10%. We also evaluate our
  approach to detect vulnerability in auto-generated code by code LLMs.
  Evaluation on a benchmark of high-risk code scenarios shows a reduction of up
  to 90% vulnerability reduction.
paper_authors._id: 647ea05c11084fb5831e6091
paper_authors.name: Aaron Chan
paper_authors._id: 647ea05c11084fb5831e6092
paper_authors.name: Anant Kharkar
paper_authors._id: 647ea05c11084fb5831e6093
paper_authors.name: Roshanak Zilouchian Moghaddam
paper_authors._id: 647ea05c11084fb5831e6094
paper_authors.name: Yevhen Mohylevskyy
paper_authors._id: 647ea05c11084fb5831e6095
paper_authors.name: Alec Helyar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-06T09:53:17.489Z
paper_authors._id: 6385710c20e00df40e4e5275
paper_authors.avatarUrl: /avatars/bd793f0f456cbb1fca47ce9aca9c2acc.svg
paper_authors.fullname: Alec Helyar
paper_authors.type: user
paper_authors.user: alech97
paper_authors._id: 647ea05c11084fb5831e6096
paper_authors.name: Eslam Kamal
paper_authors._id: 647ea05c11084fb5831e6097
paper_authors.name: Mohamed Elkamhawy
paper_authors._id: 647ea05c11084fb5831e6098
paper_authors.name: Neel Sundaresan
paper_title: Transformer-based Vulnerability Detection in Code at EditTime:
    Zero-shot, Few-shot, or Fine-tuning?
paper_summary: Software vulnerabilities bear enterprises significant costs. Despite
  extensive efforts in research and development of software vulnerability
  detection methods, uncaught vulnerabilities continue to put software owners and
  users at risk. Many current vulnerability detection methods require that code
  snippets can compile and build before attempting detection. This,
  unfortunately, introduces a long latency between the time a vulnerability is
  injected to the time it is removed, which can substantially increases the cost
  of fixing a vulnerability. We recognize that the current advances in machine
  learning can be used to detect vulnerable code patterns on syntactically
  incomplete code snippets as the developer is writing the code at EditTime. In
  this paper we present a practical system that leverages deep learning on a
  large-scale data set of vulnerable code patterns to learn complex
  manifestations of more than 250 vulnerability types and detect vulnerable code
  patterns at EditTime. We discuss zero-shot, few-shot, and fine-tuning
  approaches on state of the art pre-trained Large Language Models (LLMs). We
  show that in comparison with state of the art vulnerability detection models
  our approach improves the state of the art by 10%. We also evaluate our
  approach to detect vulnerability in auto-generated code by code LLMs.
  Evaluation on a benchmark of high-risk code scenarios shows a reduction of up
  to 90% vulnerability reduction.
paper_ai_keywords: deep learning pre-trained Large Language Models (LLMs) zero-shot few-shot fine-tuning vulnerable code patterns high-risk code scenarios