15

Efficient LLM Algorithmic Survey, Nov, 2023, USA.

Ding, Chen, et al.

Computation Efficiency. While distributed training offers potential benefits in speeding up the training of large models, it
also introduces complexities that affect scalability. A notable observation is the decrease in FLOPs per GPU when training is
distributed across multiple GPUs, as opposed to a single GPU setup. This decrease stems from the challenges in efficiently
utilizing an increasing number of computational resources. Therefore, scalability becomes a crucial element in boosting
computation efficiency during the training process, particularly in multi-GPU settings.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey