Machine Learning, Transformers, Data Science, Python, Hugging Face.
GPTQConfig( bits=4, group_size=128, dataset="c4", desc_act=False, tokenizer=tokenizer ) # Load the model from HF quant_model = AutoModelForCausalLM.from_pretrained(hf_model_repo, quantization_config=quantization_config, device_map='auto') As mentioned, this code takes about 45 minutes to run and