Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
load_in_8bit=True when loading the model (it also requires a GPU). device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model_int8 = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto', load_in_8bit=True, ) print(f"Model size: {model_int8.get_memory_footprint():,}