4.2. Pre-training
We pre-trained our models on the GitHub dataset described in Section 3, with a standard cross-entropy
next-token prediction loss for the decoder and a masked language modeling loss (Devlin et al., 2018)
for the encoder. The masked language modeling loss was essential for improving the representation
learning of the encoder. We split GitHub ﬁles by uniformly sampling pivot locations, using content
before the pivot as input to the encoder, and content after for the decoder.
Our base 1B parameter model was trained for 106 steps with a batch size of 256. Following Kaplan
et al. (2020), we adjusted the amount of training for other model sizes such that larger models are
trained more and smaller models are trained less to optimize the use of compute. However, due to
resource limitations and to make optimal use of compute, the training of our largest 41B model was
stopped early, and therefore this model was relatively undertrained compared to models at other
alphacode