You can run it on 11G mem GPU,quantize base QuIP# method, a weights-only quantization method that is able to achieve near fp16 performance using only 2 bits per weight.

url:https://github.com/Cornell-RelaxML/quip-sharp/tree/release20231203

Downloads last month: 13

Safetensors

Model size

5.11B params

Tensor type

FP16

I16

Inference Providers NEW

Text Generation

This model is not currently available via any of the supported third-party Inference Providers, and the model is not deployed on the HF Inference API.