Machine Learning, Transformers, Data Science, Python, Hugging Face.
a bunch of long examples to compare with the original model. Both inference processes were executed on a T4 GPU, the base model took about 17–19 seconds per inference while the quantized model ran in about 8 to 9 seconds per inference, a half. All code and examples are well-explained in his