Deep Learning, Artificial Intelligence, NLP, Self Supervised Learning, Quantization.
momentum and Adam optimizers are formulated as below: For 32-bit states, Momentum and Adam consume 4 and 8 bytes per parameter. That is 4 GB and 8 GB for a 1B parameter model. The proposed 8-bit non-linear quantization is to reduce these costs to 1 GB and 2 GB. 1.2. Non-Linear Quantization