can scale almost as well as expensive Infiniband up to 2000 GPUs, which makes pretraining even more
democratizable.
Carbon Footprint of Pretraining. Following preceding research (Bender et al., 2021a; Patterson et al., 2021;
Wu et al., 2022; Dodge et al., 2022) and using power consumption estimates of GPU devices and carbon
efficiency, we aim to calculate the carbon emissions resulting from the pretraining of Llama 2 models. The
actual power usage of a GPU is dependent on its utilization and is likely to vary from the Thermal Design
Power (TDP) that we employ as an estimation for GPU power. It is important to note that our calculations
do not account for further power demands, such as those from interconnect or non-GPU server power
consumption, nor from datacenter cooling systems. Additionally, the carbon output related to the production
of AI hardware, like GPUs, could add to the overall carbon footprint as suggested by Gupta et al. (2022b,a).
Llama2