3. Training Infrastructure
We trained Gemini models using TPUv5e and TPUv4 (Jouppi et al., 2023), depending on their sizes
and configuration. Training Gemini Ultra used a large fleet of TPUv4 accelerators across multiple
datacenters. This represents a significant increase in scale over our prior flagship model PaLM-2
which presented new infrastructure challenges. Scaling up the number of accelerators results in a
proportionate decrease in the mean time between failure of hardware in the overall system. We
minimized the rate of planned reschedules and preemptions, but genuine machine failures are
commonplace across all hardware accelerators at such large scales, due to external factors such as
cosmic rays (Michalak et al., 2012).
gemini_1_report