Machine Learning, Transformers, Data Science, Python, Hugging Face.
adoption of a mixed int4/fp16 quantization scheme. Here, model weights are quantized as int4, while activations are retained in float16. During inference, weights are dynamically dequantized, and actual computations are performed in float16. This innovative approach promises memory savings close to