lavawolfiee
/

Mixtral-8x7B-Instruct-v0.1-offloading-demo

Text Generation

Transformers

Safetensors

mixtral

text-generation-inference

conversational

8-bit precision

Model card Files Files and versions Community

Attention quantization: HQQ 4-bit, groupsize 64, compress zero, compress scale with groupsize 256
Experts quantization: HQQ 2-bit, groupsize 16, compress zero, compress scale with groupsize 128

Downloads last month: 1,292

Safetensors

Model size

17.9B params

Tensor type

F16