Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
compare the distribution of our weights (same scale) and 2/ actually run the models. Indeed, PyTorch doesn’t allow INT8 matrix multiplication by default. In a real scenario, we would dequantize them to run the model (in FP16 for example) but store them as INT8. In the next section, we will use the