Deep Learning, Artificial Intelligence, NLP, Self Supervised Learning, Quantization.
variance of roughly one both at initialization and during training. Additionally, the uniform distribution initialization has less extreme values than a normal distribution, reducing maximum gradient size. 3. Results NLP & CV Tasks 8-bit optimizers match replicated 32-bit performance for all tasks.