Machine Learning, Transformers, Data Science, Python, Hugging Face.
consumes a peak of 32 GB of GPU VRAM. “You will need a GPU to quantize a model. We will put the model in the CPU and move the modules back and forth to the GPU in order to quantize them. If you want to maximize your GPUs usage while using CPU offload, you can set device_map = "auto" “ [6], hugging