Quantization Script
#1
by
GusPuffy
- opened
Hello, I am having issues quantizing 70b models, would you be able to provide the script and list the hardware or service provider you used to quantize the model? I have rented h100s on runpod and still been unable to quantize a 70b with AWQ. Thank you!
Hi GusPuffy,
I quantized this in-house (no provider), and the code/hardware I used is in the documentation. The memory and GPU settings in the documentation are specific to this hardware (dual RTX 3090s), so you need to adjust them for an H100.
Do you need specific help building a quantization setup that works on an H100 RunPod, or are you looking to get a particular 70B model AWQ quantized?
Given it has been 4 days, I am closing this as resolved.
ibnzterrell
changed discussion status to
closed