scenarios (23 in C and 17 in Python).
Pearce et al. (2022) had previously evaluated the security of GitHub Copilot (as of August 2021), and
in this paper, we use the same methodology to evaluate StarCoderBase, InCoder-6B, CodeGen-16B-
Multi, and OpenAI’s code-cushman-001. We use the original benchmarking methodology: generating
25 completions per scenario at temperature 0.2 (1,000 completions per model). The dataset supports
fill-in-the-middle, so we include this configuration on models that support it. The results are shown
in Table 16; Valid gives the percentage of solutions that were syntactically valid (using py compile
for Python and gcc for C), and Insecure shows the percentage of valid solutions that contained the
vulnerability the scenario tests for. From this table, we draw the following conclusions.
StarCoder_paper (1)