title: SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
summary: Existing works have established multiple benchmarks to highlight the security
  risks associated with Code GenAI. These risks are primarily reflected in two
  areas: a model potential to generate insecure code (insecure coding) and its
  utility in cyberattacks (cyberattack helpfulness). While these benchmarks have
  made significant strides, there remain opportunities for further improvement.
  For instance, many current benchmarks tend to focus more on a model ability to
  provide attack suggestions rather than its capacity to generate executable
  attacks. Additionally, most benchmarks rely heavily on static evaluation
  metrics, which may not be as precise as dynamic metrics such as passing test
  cases. Conversely, expert-verified benchmarks, while offering high-quality
  data, often operate at a smaller scale. To address these gaps, we develop
  SecCodePLT, a unified and comprehensive evaluation platform for code GenAIs'
  risks. For insecure code, we introduce a new methodology for data creation that
  combines experts with automatic generation. Our methodology ensures the data
  quality while enabling large-scale generation. We also associate samples with
  test cases to conduct code-related dynamic evaluation. For cyberattack
  helpfulness, we set up a real environment and construct samples to prompt a
  model to generate actual attacks, along with dynamic metrics in our
  environment. We conduct extensive experiments and show that SecCodePLT
  outperforms the state-of-the-art (SOTA) benchmark CyberSecEval in security
  relevance. Furthermore, it better identifies the security risks of SOTA models
  in insecure coding and cyberattack helpfulness. Finally, we apply SecCodePLT to
  the SOTA code agent, Cursor, and, for the first time, identify non-trivial
  security risks in this advanced coding agent.
paper_authors._id: 670f11460dd9c30426f5eb0a
paper_authors.name: Yu Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:02.861Z
paper_authors._id: 62f82e52870a3f98bbf9e302
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f82e52870a3f98bbf9e302/5pN3oNBouZWlYu-uKa7lA.jpeg
paper_authors.fullname: Yu Yang
paper_authors.type: user
paper_authors.user: yuyangy
paper_authors._id: 670f11460dd9c30426f5eb0b
paper_authors.name: Yuzhou Nie
paper_authors._id: 670f11460dd9c30426f5eb0c
paper_authors.name: Zhun Wang
paper_authors._id: 670f11460dd9c30426f5eb0d
paper_authors.name: Yuheng Tang
paper_authors._id: 670f11460dd9c30426f5eb0e
paper_authors.name: Wenbo Guo
paper_authors._id: 670f11460dd9c30426f5eb0f
paper_authors.name: Bo Li
paper_authors._id: 670f11460dd9c30426f5eb10
paper_authors.name: Dawn Song
paper_title: SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
paper_summary: Existing works have established multiple benchmarks to highlight the security
  risks associated with Code GenAI. These risks are primarily reflected in two
  areas: a model potential to generate insecure code (insecure coding) and its
  utility in cyberattacks (cyberattack helpfulness). While these benchmarks have
  made significant strides, there remain opportunities for further improvement.
  For instance, many current benchmarks tend to focus more on a model ability to
  provide attack suggestions rather than its capacity to generate executable
  attacks. Additionally, most benchmarks rely heavily on static evaluation
  metrics, which may not be as precise as dynamic metrics such as passing test
  cases. Conversely, expert-verified benchmarks, while offering high-quality
  data, often operate at a smaller scale. To address these gaps, we develop
  SecCodePLT, a unified and comprehensive evaluation platform for code GenAIs'
  risks. For insecure code, we introduce a new methodology for data creation that
  combines experts with automatic generation. Our methodology ensures the data
  quality while enabling large-scale generation. We also associate samples with
  test cases to conduct code-related dynamic evaluation. For cyberattack
  helpfulness, we set up a real environment and construct samples to prompt a
  model to generate actual attacks, along with dynamic metrics in our
  environment. We conduct extensive experiments and show that SecCodePLT
  outperforms the state-of-the-art (SOTA) benchmark CyberSecEval in security
  relevance. Furthermore, it better identifies the security risks of SOTA models
  in insecure coding and cyberattack helpfulness. Finally, we apply SecCodePLT to
  the SOTA code agent, Cursor, and, for the first time, identify non-trivial
  security risks in this advanced coding agent.
paper_ai_keywords: Code GenAI insecure code cyber work benchmarks attack suggestions dynamic metrics expert-verified benchmarks test cases code generation SecCodePLT unified evaluation platform data creation automatic generation large-scale generation dynamic evaluation security risks state-of-the-art (SOTA) benchmark CyberSecEval Cursor non-trivial security risks security relevance insecure coding cyberattack helpfulness