|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
pipeline_tag: text-generation |
|
|
base_model: |
|
|
- Qwen/Qwen3-1.7B |
|
|
tags: |
|
|
- neuralmagic |
|
|
- redhat |
|
|
- llmcompressor |
|
|
- quantized |
|
|
- FP8 |
|
|
--- |
|
|
|
|
|
# Qwen3-1.7B-FP8-dynamic |
|
|
|
|
|
## Model Overview |
|
|
- **Model Architecture:** Qwen3ForCausalLM |
|
|
- **Input:** Text |
|
|
- **Output:** Text |
|
|
- **Model Optimizations:** |
|
|
- **Activation quantization:** FP8 |
|
|
- **Weight quantization:** FP8 |
|
|
- **Intended Use Cases:** |
|
|
- Reasoning. |
|
|
- Function calling. |
|
|
- Subject matter experts via fine-tuning. |
|
|
- Multilingual instruction following. |
|
|
- Translation. |
|
|
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). |
|
|
- **Release Date:** 05/02/2025 |
|
|
- **Version:** 1.0 |
|
|
- **Model Developers:** RedHat (Neural Magic) |
|
|
|
|
|
### Model Optimizations |
|
|
|
|
|
This model was obtained by quantizing activations and weights of [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) to FP8 data type. |
|
|
This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). |
|
|
Weight quantization also reduces disk size requirements by approximately 50%. |
|
|
|
|
|
Only weights and activations of the linear operators within transformers blocks are quantized. |
|
|
Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. |
|
|
The [llm-compressor](https://github.com/vllm-project/llm-compressor) library is used for quantization. |
|
|
|
|
|
|
|
|
## Deployment |
|
|
|
|
|
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below. |
|
|
|
|
|
```python |
|
|
from vllm import LLM, SamplingParams |
|
|
from transformers import AutoTokenizer |
|
|
|
|
|
model_id = "RedHatAI/Qwen3-1.7B-FP8-dynamic" |
|
|
number_gpus = 1 |
|
|
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256) |
|
|
|
|
|
messages = [ |
|
|
{"role": "user", "content": prompt} |
|
|
] |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
|
|
messages = [{"role": "user", "content": "Give me a short introduction to large language model."}] |
|
|
|
|
|
prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
|
|
|
|
|
llm = LLM(model=model_id, tensor_parallel_size=number_gpus) |
|
|
|
|
|
outputs = llm.generate(prompts, sampling_params) |
|
|
|
|
|
generated_text = outputs[0].outputs[0].text |
|
|
print(generated_text) |
|
|
``` |
|
|
|
|
|
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details. |
|
|
|
|
|
## Creation |
|
|
|
|
|
<details> |
|
|
<summary>Creation details</summary> |
|
|
This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. |
|
|
|
|
|
|
|
|
```python |
|
|
from llmcompressor.modifiers.quantization import QuantizationModifier |
|
|
from llmcompressor.transformers import oneshot |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Load model |
|
|
model_stub = "Qwen/Qwen3-1.7B" |
|
|
model_name = model_stub.split("/")[-1] |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_stub) |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_stub) |
|
|
|
|
|
# Configure the quantization algorithm and scheme |
|
|
recipe = QuantizationModifier( |
|
|
ignore=["lm_head"], |
|
|
targets="Linear", |
|
|
scheme="FP8_dynamic", |
|
|
) |
|
|
|
|
|
# Apply quantization |
|
|
oneshot( |
|
|
model=model, |
|
|
recipe=recipe, |
|
|
) |
|
|
|
|
|
# Save to disk in compressed-tensors format |
|
|
save_path = model_name + "-FP8-dynamic" |
|
|
model.save_pretrained(save_path) |
|
|
tokenizer.save_pretrained(save_path) |
|
|
print(f"Model and tokenizer saved to: {save_path}") |
|
|
``` |
|
|
</details> |
|
|
|
|
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on reasoning tasks using [lighteval](https://github.com/neuralmagic/lighteval/tree/reasoning). |
|
|
[vLLM](https://docs.vllm.ai/en/stable/) was used for all evaluations. |
|
|
|
|
|
<details> |
|
|
<summary>Evaluation details</summary> |
|
|
|
|
|
**lm-evaluation-harness** |
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks openllm \ |
|
|
--apply_chat_template\ |
|
|
--fewshot_as_multiturn \ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks mgsm \ |
|
|
--apply_chat_template\ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=16384,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks leaderboard \ |
|
|
--apply_chat_template\ |
|
|
--fewshot_as_multiturn \ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
**lighteval** |
|
|
|
|
|
lighteval_model_arguments.yaml |
|
|
```yaml |
|
|
model_parameters: |
|
|
model_name: RedHatAI/Qwen3-1.7B-FP8-dynamic |
|
|
dtype: auto |
|
|
gpu_memory_utilization: 0.9 |
|
|
max_model_length: 40960 |
|
|
generation_parameters: |
|
|
temperature: 0.6 |
|
|
top_k: 20 |
|
|
min_p: 0.0 |
|
|
top_p: 0.95 |
|
|
max_new_tokens: 32768 |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|aime24|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|aime25|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|math_500|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|gpqa:diamond|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks extended|lcb:codegeneration \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
</details> |
|
|
|
|
|
### Accuracy |
|
|
|
|
|
<table> |
|
|
<tr> |
|
|
<th>Category |
|
|
</th> |
|
|
<th>Benchmark |
|
|
</th> |
|
|
<th>Qwen3-1.7B |
|
|
</th> |
|
|
<th>Qwen3-1.7B-FP8-dynamic<br>(this model) |
|
|
</th> |
|
|
<th>Recovery |
|
|
</th> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="7" ><strong>OpenLLM v1</strong> |
|
|
</td> |
|
|
<td>MMLU (5-shot) |
|
|
</td> |
|
|
<td>56.82 |
|
|
</td> |
|
|
<td>56.02 |
|
|
</td> |
|
|
<td>98.6% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>ARC Challenge (25-shot) |
|
|
</td> |
|
|
<td>43.00 |
|
|
</td> |
|
|
<td>42.83 |
|
|
</td> |
|
|
<td>99.6% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GSM-8K (5-shot, strict-match) |
|
|
</td> |
|
|
<td>43.67 |
|
|
</td> |
|
|
<td>41.47 |
|
|
</td> |
|
|
<td>95.0% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Hellaswag (10-shot) |
|
|
</td> |
|
|
<td>48.08 |
|
|
</td> |
|
|
<td>48.11 |
|
|
</td> |
|
|
<td>100.1% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Winogrande (5-shot) |
|
|
</td> |
|
|
<td>58.01 |
|
|
</td> |
|
|
<td>57.70 |
|
|
</td> |
|
|
<td>99.5% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>TruthfulQA (0-shot, mc2) |
|
|
</td> |
|
|
<td>49.35 |
|
|
</td> |
|
|
<td>48.60 |
|
|
</td> |
|
|
<td>98.5% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><strong>Average</strong> |
|
|
</td> |
|
|
<td><strong>49.82</strong> |
|
|
</td> |
|
|
<td><strong>49.12</strong> |
|
|
</td> |
|
|
<td><strong>98.6%</strong> |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="7" ><strong>OpenLLM v2</strong> |
|
|
</td> |
|
|
<td>MMLU-Pro (5-shot) |
|
|
</td> |
|
|
<td>23.45 |
|
|
</td> |
|
|
<td>21.38 |
|
|
</td> |
|
|
<td>91.1% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>IFEval (0-shot) |
|
|
</td> |
|
|
<td>71.08 |
|
|
</td> |
|
|
<td>70.93 |
|
|
</td> |
|
|
<td>99.8% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>BBH (3-shot) |
|
|
</td> |
|
|
<td>7.13 |
|
|
</td> |
|
|
<td>5.41 |
|
|
</td> |
|
|
<td>--- |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Math-lvl-5 (4-shot) |
|
|
</td> |
|
|
<td>35.91 |
|
|
</td> |
|
|
<td>34.71 |
|
|
</td> |
|
|
<td>96.7% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GPQA (0-shot) |
|
|
</td> |
|
|
<td>0.11 |
|
|
</td> |
|
|
<td>0.00 |
|
|
</td> |
|
|
<td>--- |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MuSR (0-shot) |
|
|
</td> |
|
|
<td>7.97 |
|
|
</td> |
|
|
<td>7.18 |
|
|
</td> |
|
|
<td>--- |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><strong>Average</strong> |
|
|
</td> |
|
|
<td><strong>24.28</strong> |
|
|
</td> |
|
|
<td><strong>23.27</strong> |
|
|
</td> |
|
|
<td><strong>95.8%</strong> |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><strong>Multilingual</strong> |
|
|
</td> |
|
|
<td>MGSM (0-shot) |
|
|
</td> |
|
|
<td>22.10 |
|
|
</td> |
|
|
<td> |
|
|
</td> |
|
|
<td> |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="6" ><strong>Reasoning<br>(generation)</strong> |
|
|
</td> |
|
|
<td>AIME 2024 |
|
|
</td> |
|
|
<td>43.96 |
|
|
</td> |
|
|
<td>40.10 |
|
|
</td> |
|
|
<td>91.2% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>AIME 2025 |
|
|
</td> |
|
|
<td>32.29 |
|
|
</td> |
|
|
<td>32.29 |
|
|
</td> |
|
|
<td>100.0% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GPQA diamond |
|
|
</td> |
|
|
<td>38.38 |
|
|
</td> |
|
|
<td>38.89 |
|
|
</td> |
|
|
<td>101.3% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Math-lvl-5 |
|
|
</td> |
|
|
<td>89.00 |
|
|
</td> |
|
|
<td>88.80 |
|
|
</td> |
|
|
<td>99.8% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>LiveCodeBench |
|
|
</td> |
|
|
<td>33.44 |
|
|
</td> |
|
|
<td> |
|
|
</td> |
|
|
<td> |
|
|
</td> |
|
|
</tr> |
|
|
</table> |
