6.2.2 Mixed Precision Training. In the realm of LLM pre-training, mixed precision training emerges as a critical strategy
for enhancing both memory and computational efficiency. Traditionally, neural network training involves storing weights,
gradients, and activations in full-precision (FP32) format. However, for extremely large models, this approach can be resource-
intensive. To address this, reduced-precision formats like FP16 or INT8 are adopted. These formats not only reduce memory
usage but also expedite communication processes within the model. In addition, modern GPUs are typically more adept at
handling FP16 computations compared to FP32, offering a further boost in computational speed.

16

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Efficient LLM Algorithmic Survey, Nov, 2023, USA.

Table 2. Summary of different parallelism strategies for efficiency.

Parallelism Strategy

Data Parallelism (DP)

Model Parallelism (MP)

Tensor Parallelism (TP)
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey