Deep Learning, Artificial Intelligence, NLP, Self Supervised Learning, Quantization.
The proposed 8-bit optimizers save up to 8.5 GB of GPU memory for the largest 1.5B parameter language model and 2.0 GB for RoBERTa. The models are now accessible with smaller GPUs. Ablation Study The Ablations show that dynamic quantization, block-wise quantization, and the stable embedding layer