Deep Learning, Artificial Intelligence, NLP, Self Supervised Learning, Quantization.
optimizer update with 8-bit states is straightforward. the 8-bit optimizer states are dequantized to 32-bit to perform the update, and then the states are quantized back to 8-bit for storage. 2.1. Block-wise Quantization Block-wise dynamic quantization reduces this cost by chunking an input tensor