Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
dtype=torch.int8) The difference between the original (FP32) and quantized values (INT8) is clear, but the difference between absmax and zero-point weights is more subtle. In this case, the inputs look shifted by a value of -1. This suggests that the weight distribution in this layer is quite