Machine Learning, Transformers, Data Science, Python, Hugging Face.
the inference latency is on par with FP16 inference. AutoGPTQ supports Exllama kernels for a wide range of architectures. The integration comes with native RoCm support for AMD GPUs. Finetuning with PEFT is available. Extracted from the Huggingface blog article “Making LLMs lighter with AutoGPTQ