|
|
--- |
|
|
base_model: shisa-ai/shisa-v2-llama3.1-405b |
|
|
datasets: |
|
|
- shisa-ai/shisa-v2-sharegpt |
|
|
- shisa-ai/deepseekv3-ultrafeedback-armorm-dpo |
|
|
language: |
|
|
- ja |
|
|
- en |
|
|
- ko |
|
|
- zh |
|
|
library_name: transformers |
|
|
license: llama3.1 |
|
|
model_name: shisa-v2-llama3.1-405b |
|
|
quantized_by: leonardlin |
|
|
--- |
|
|
|
|
|
## About |
|
|
This repo contains select GGUF quants of [shisa-ai/shisa-v2-llama3.1-405b](https://huggingface.co/shisa-ai/shisa-v2-llama3.1-405b) |
|
|
- All quants were created with `b5503` of upstream [llama.cpp](https://github.com/ggerganov/llama.cpp) |
|
|
- All quants are weighted/imatrix quants created from our [shisa-ai/shisa-v2-sharegpt](https://huggingface.co/datasets/shisa-ai/shisa-v2-sharegpt) bilingual dataset on the fp16 model except for the Q8_0 |
|
|
- Files are pre-split at 45GB (below HF's 50GB upload limit). Modern llama.cpp builds should be able to load the sequential files automatically, but you can use `llama-gguf-split --merge` if you want to merge them back together |
|
|
|
|
|
## Provided Quants |
|
|
|
|
|
| Type | Size (GiB) | |
|
|
|:--------|----------:| |
|
|
| IQ2_XXS | 100 | |
|
|
| IQ3_XS | 155 | |
|
|
| IQ3_M | 170 | |
|
|
| IQ4_XS | 202 | |
|
|
| Q4_K_M | 227 | |
|
|
| Q8_0 | 402 | |
|
|
|
|
|
|
|
|
## Quant Quality |
|
|
All quants have been tested with JA MT-Bench (judged by GPT-4.1) as a rough guide for quality: |
|
|
|
|
|
| Quant | Size (GiB)| % Diff | Overall | Writing | Roleplay | Reasoning | Math | Coding | Extraction | STEM | Humanities | |
|
|
|--------------|--------:|-------------:|---------:|----------:|---------:|----------:|---------:|---------:|-----------:|---------:|-----------:| |
|
|
| Full FP16 | 810 | | **9.13** | 9.25 | **9.55** | 8.15 | 8.90 | 9.10 | 9.65 | 9.10 | 9.35 | |
|
|
| IQ3_M | 170 | -0.99 | 9.04 | 8.90 | 9.45 | 7.75 | 8.95 | 8.95 | 9.70 | **9.15** | 9.50 | |
|
|
| Q4_K_M | 227 | -1.10 | 9.03 | **9.40** | 9.00 | 8.25 | 8.85 | **9.10** | 9.50 | 8.90 | 9.25 | |
|
|
| Q8_0 | 405 | -1.20 | 9.02 | **9.40** | 9.05 | **8.30** | **9.20** | 8.70 | 9.50 | 8.45 | 9.55 | |
|
|
| W8A8-INT8 | 405 | -1.42 | 9.00 | 9.20 | 9.35 | 7.80 | 8.75 | 9.00 | 9.80 | 8.65 | 9.45 | |
|
|
| FP8-Dynamic | 405 | -3.29 | 8.83 | 8.70 | 9.20 | 7.85 | 8.80 | 8.65 | 9.30 | 8.80 | 9.35 | |
|
|
| IQ3_XS | 155 | -3.50 | 8.81 | 8.70 | 9.05 | 7.70 | 8.60 | 8.95 | 9.35 | 8.70 | 9.45 | |
|
|
| IQ4_XS | 202 | -3.61 | 8.80 | 8.85 | **9.55** | 6.90 | 8.35 | 8.60 | **9.90** | 8.65 | **9.60** | |
|
|
| *70B FP16* | 140 | -7.89 | 8.41 | 7.95 | 9.05 | 6.25 | 8.30 | 8.25 | 9.70 | 8.70 | 9.05 | |
|
|
| IQ2_XXS | 100 | -18.18 | 7.47 | 7.50 | 6.80 | 5.15 | 7.55 | 7.30 | 9.05 | 7.65 | 8.80 | |
|
|
|
|
|
Due to margin of error, you could probably fairly say that the IQ3_M, Q4_K_M, and Q8_0 GGUFs have almost no functional loss versus the FP16. |
|
|
|
|
|
Interestingly enough, while roleplay takes one of the biggest hits, writing seems to be improved on the Q4 and Q8? I think you'd really need to test more (more samples, more runs, more evals) to really see what's going on. Also interestingly the XS quants track pretty consistently, with the IQ4_XS doing worse than the IQ3_M. |
|
|
|
|
|
The IQ2_XXS scores extremely poorly. I included the 70B Full FP16 scores as a baseline and I'd expect you'd be better off running a decent Shisa V2 70B Q4_K_M (40GB) or IQ3_M (32GB) vs the IQ2. |
|
|
|
|
|
In an ideal world, of course, you should test different quants on your downstream tasks, but I understand that that's not always an option. Based on this testing though, if you had to pick on bang/buck quant blind, I'd start with the IQ3_M. |
|
|
|
|
|
|
|
|
## Making Quants |
|
|
``` |
|
|
# first you need an fp16 - setup llama.cpp python env and run something like |
|
|
python convert_hf_to_gguf.py ~/.cache/huggingface/hub/models--shisa-ai--shisa-v2-llama3.1-405b/snapshots/71b83a7cb998c3a44f59c83a9928596ac348b9b5 --outfile shisa-v2-llama3.1-405b-fp16.gguf |
|
|
|
|
|
# Create imatrix: using 4 x H200 you can load 88 layers, takes about 1h15m |
|
|
CUDA_VISIBLE_DEVICES=4,5,6,7 build/bin/llama-imatrix -m shisa-v2-llama3.1-405b-fp16.gguf -f /data/quantize/shisa-v2-llama-3.1-405b/gguf/calibration_chat.txt -o imatrix.dat -c 512 -b 512 --chunks 100 -ngl 88 |
|
|
|
|
|
# create your imatrix quants |
|
|
build/bin/llama-quantize --imatrix imatrix.dat shisa-v2-llama3.1-405b-fp16.gguf shisa-v2-llama3.1-405b-IQ3_XS.gguf IQ3_XS |
|
|
|
|
|
# split the quants |
|
|
build/bin/llama-gguf-split --split-max-size 45G shisa-v2-llama3.1-405b-IQ3_XS.gguf shisa-v2-llama3.1-405b-IQ3_XS |
|
|
|
|
|
# upload (bash loop) |
|
|
for f in shisa-v2-llama3.1-405b-IQ3_XS-0000*; do huggingface-cli upload shisa-ai/shisa-v2-llama3.1-405b-GGUF "$f"; done |
|
|
``` |
