Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?
<|reference_start|>Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?: Background: Software Vulnerability (SV) prediction needs large-sized and high-quality data to perform well. Current SV datasets mostly require expensive labeling efforts by experts (human-labeled) and thus are limited in size. Meanwhile, there are growing efforts in automatic SV labeling at scale. However, the fitness of auto-labeled data for SV prediction is still largely unknown. Aims: We quantitatively and qualitatively study the quality and use of the state-of-the-art auto-labeled SV data, D2A, for SV prediction. Method: Using multiple sources and manual validation, we curate clean SV data from human-labeled SV-fixing commits in two well-known projects for investigating the auto-labeled counterparts. Results: We discover that 50+% of the auto-labeled SVs are noisy (incorrectly labeled), and they hardly overlap with the publicly reported ones. Yet, SV prediction models utilizing the noisy auto-labeled SVs can perform up to 22% and 90% better in Matthews Correlation Coefficient and Recall, respectively, than the original models. We also reveal the promises and difficulties of applying noise-reduction methods for automatically addressing the noise in auto-labeled SV data to maximize the data utilization for SV prediction. Conclusions: Our study informs the benefits and challenges of using auto-labeled SVs, paving the way for large-scale SV prediction.<|reference_end|>
@article{le2024automatic,
    title={Automatic Data Labeling for Software Vulnerability Prediction Models:
  How Far Are We?},
    author={Triet H. M. Le, M. Ali Babar},
    journal={arXiv preprint arXiv:2407.17803},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.17803},
    primaryClass={cs.SE cs.CR cs.LG}
}