Another innovative approach is Activation Compressed Training (ACT) [33], which focuses on compressing activations to
an average of 2 bits across multiple tasks. ACT computes gradients using a compressed version of activations saved during
the backward process, leading to a significant reduction in memory requirements for activations. This compression enables
training with substantially larger batch sizes, ranging from 6.6× to 14× larger than traditional methods. Overall, mixed
precision training stands as a testament to the evolving landscape of LLM training, where efficiency and performance are
continually balanced through innovative techniques.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey