<NAME>, <EMAIL>, <KEY>, <PASSWORD>

To mask IP addresses, we randomly selected an IP address from 5 synthetic, private, non-internet-
facing IP addresses of the same type that can be found in Appendix C.

Compute resources We used the PII detection model to identify PII across all programming
languages in the training dataset, including GitHub issues, Git commits, and Jupyter notebooks. The
total dataset amounts to 815 GB in size. We ran inference on multiple NVIDIA A100 80 GB GPUs,
which required 800 GPU-hours.

5 M O D E L T R A I N I N G

This section presents information on the training process of the StarCoder models. Before we proceed,
we first clarify the differences between the two models:

StarCoderBase is the first model trained on 1 trillion tokens sourced from the curated dataset

StarCoder is the fine-tuned version of StarCoderBase, trained on another 35B Python tokens

described in Section 3.

(roughly 2 epochs).
StarCoder_paper (1)