|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
- zh |
|
|
- es |
|
|
base_model: |
|
|
- shuttleai/shuttle-3.5 |
|
|
tags: |
|
|
- Qwen |
|
|
- Shuttle |
|
|
- GGUF |
|
|
- 32b |
|
|
- quantized |
|
|
- Q8_0 |
|
|
--- |
|
|
|
|
|
# Shuttle 3.5 — Q8_0 GGUF Quant |
|
|
|
|
|
This repo contains a GGUF quantized version of [ShuttleAI's Shuttle 3.5 model](https://huggingface.co/shuttleai/shuttle-3.5), a high-performance instruction-tuned variant of Qwen 3 32B. This quant was built for efficient local inference without sacrificing quality. |
|
|
|
|
|
## 🔗 Base Model |
|
|
|
|
|
- **Original**: [shuttleai/shuttle-3.5](https://huggingface.co/shuttleai/shuttle-3.5) |
|
|
- **Parent architecture**: Qwen 3 32B |
|
|
- **Quantized by**: Lex-au |
|
|
- **Quantization format**: GGUF Q8_0 |
|
|
|
|
|
## 📦 Model Size |
|
|
|
|
|
| Format | Size | |
|
|
|----------|----------| |
|
|
| Original (safetensors, F16) | 65.52 GB | |
|
|
| Q8_0 (GGUF) | 34.8 GB | |
|
|
|
|
|
**Compression Ratio**: ~47% |
|
|
**Size Reduction**: ~18% absolute (30.72 GB saved) |
|
|
|
|
|
## 🧪 Quality |
|
|
|
|
|
- Q8_0 is **near-lossless**, preserving almost all performance of the full-precision model. |
|
|
- Ideal for high-quality inference on capable consumer hardware. |
|
|
|
|
|
## 🚀 Usage |
|
|
|
|
|
Compatible with all major GGUF-supporting runtimes, including: |
|
|
|
|
|
- `llama.cpp` |
|
|
- `KoboldCPP` |
|
|
- `text-generation-webui` |
|
|
- `llamafile` |
|
|
- `LM Studio` |
|
|
|
|
|
Example with `llama.cpp`: |
|
|
|
|
|
```bash |
|
|
./main -m shuttle-3.5.Q8_0.gguf --ctx-size 4096 --threads 16 --prompt "Describe the effects of quantum decoherence in plain English." |
|
|
|
