The largest model in LaMDA was pre-trained on 1024 TPU-V3 chips and 123 TFLOPS/s for 57.7 days with FLOPS
utilization of 56.5% using GSPMD [95]. The total FLOPS is 56.5% * 123 TFLOPS/s * 1024 chips * 57.7 days
= 3.55E+23, which is higher than 3.14E+23, corresponding to the total FLOPS of GPT-3 [12]. The PUE of our
datacenter is 1.10, and Measured System Average Power per Accelerator for our experiment on TPUv3 is roughly
289W (borrowing Meena measurements from [108]), which means the total energy cost of our model is 57.7 days *
1024 chips * 289W * 1.1 * 24 hours/day = 451 MWh, 0.4X the energy of GPT-3 [12, 108]. At the time of training, our
energy mix (kg CO2e per kWh) is around 0.056, so the total carbon footprint of LaMDA’s pre-training of the largest
model is approximately 25.2 tCO2e. The carbon footprint of pre-training of smaller models and ﬁne-tuning of all
models is approximately 0.7 tCO2e (see Table 27), which brings the total footprint of LaMDA to approximately 26
LaMDA- Language Models for Dialog Applications