Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
{ppl.item():.2f}") Perplexity (original): 15.53 Perplexity (LLM.int8()): 7.93 In this case, the perplexity of the quantized model is twice as low as the original one. In general, this is not the case, but it shows that this quantization technique is very competitive. In fact, the authors of