Large Language Models, Artificial Intelligence, Data Science, Machine Learning, Editors Pick.
use it here (it will probably be fixed in the future, however). # Load quantize config, model and tokenizer quantize_config = BaseQuantizeConfig( bits=4, group_size=128, damp_percent=0.01, desc_act=False, ) model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config) tokenizer =