No pre-training

GitHub (Python only)

MassiveText

GitHub (all languages)

Solve rate

10@1K 10@10K 10@100K
10.5%
4.5%
15.5%
5.8%
21.2%
9.7%
12.4%
21.5%

7.0%
11.1%
16.1%
17.3%

Table 7 | Model solve rate with diﬀerent pre-training settings and datasets.

5.3.4. Model enhancements
As discussed in Section 4, we adopted training and model enhancements which signiﬁcantly improved
the solve rate relative to the standard encoder-decoder transformer setup. Table 8 presents the results
of a build-up ablation of the enhancements we added to AlphaCode, starting from the base setting
with no enhancements (beyond the multi-query attention change discussed in Section 5.3.2). We
added one new setting at a time, with the ﬁnal setting that corresponds to AlphaCode reported at the
bottom of the table. Each additional setting improves performance and combining the 5 enhancements
together increases the 10@100k solve rate from 15.2% to 24.1%, although the contribution depends
on the number of samples.
alphacode