|
|
--- |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- fp16 |
|
|
- dequantized |
|
|
- gpt-oss |
|
|
- mxfp4-upcast |
|
|
base_model: openai/gpt-oss-120b |
|
|
model-index: |
|
|
- name: gpt-oss-120b-fp16 |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
# |
|
|
|
|
|
## Precision: FP32 vs FP16 (and BF16) |
|
|
|
|
|
This project saves dequantized checkpoints in **FP16** (bf16 -> fp16) |
|
|
|
|
|
- **FP32 (single precision, 32-bit, 4 bytes/param)** |
|
|
Reference/default precision in many frameworks. Highest numerical range/precision, **largest memory**. |
|
|
- **FP16 (half precision, 16-bit, 2 bytes/param)** |
|
|
Half the memory of FP32. Great for **inference** on modern GPUs; may underflow/overflow more easily than BF16. |
|
|
- **BF16 (bfloat16, 16-bit, 2 bytes/param)** |
|
|
Same memory as FP16, **wider exponent like FP32**, often more numerically robust than FP16; slightly less precision in mantissa. |
|
|
|
|
|
> In this repo, output precision is **FP16** (default) or **BF16** via `--dtype`. |
|
|
> **FP32 output is not offered** because it doubles disk/RAM vs FP16/BF16 with minimal inference benefit on modern hardware. |
|
|
|
|
|
### Memory math (example: 120B parameters) |
|
|
|
|
|
Each parameter stores one number: |
|
|
|
|
|
| Format | Bits | Bytes/param | Approx size for 120B params | |
|
|
|-------:|-----:|-------------:|-----------------------------:| |
|
|
| FP32 | 32 | 4 | ~ **447 GiB** | |
|
|
| FP16 | 16 | 2 | ~ **224 GiB** | |
|
|
| BF16 | 16 | 2 | ~ **224 GiB** | |
|
|
|
|
|
> Calculation (GiB): `params * bytes_per_param / 1024^3` |
|
|
> For 120,000,000,000 params: |
|
|
> FP32: 480e9 B ≈ 447.03 GiB |
|
|
> FP16/BF16: 240e9 B ≈ 223.52 GiB |
|
|
|
|
|
### When to use which |
|
|
|
|
|
- **Inference on modern NVIDIA GPUs (Turing+/Ampere+/Ada/Hopper):** |
|
|
Use **FP16** (default here) or **BF16**. You’ll get large memory savings and typically **equal or faster** throughput than FP32 thanks to tensor cores. |
|
|
|
|
|
- **Training / Finetuning:** |
|
|
Use **mixed precision** (BF16 or FP16 compute with an FP32 master copy of weights/optimizer states). |
|
|
If your GPU supports BF16 well (e.g., A100/H100), **BF16** is preferred for numeric stability. |
|
|
(This tool focuses on exporting dequantized checkpoints, not training loops.) |
|
|
|
|
|
- **If you hit numeric issues in FP16:** |
|
|
Try **BF16** (`--dtype bf16`). Same size as FP16 but usually more stable due to FP32-like exponent range. |
|
|
|
|
|
### Notes |
|
|
|
|
|
- **FP32** remains the gold standard for numeric headroom and deterministic baselines, but for **inference** it’s typically unnecessary and **costly** (2× memory vs FP16/BF16). |
|
|
- **Tensor cores** accelerate FP16/BF16 GEMMs on most modern NVIDIA GPUs; FP32 is often slower and more memory-bound. |
|
|
- If a downstream runtime expects a specific dtype, export to that: FP16 for speed/memory, BF16 for robustness. |
|
|
|
|
|
--- |
|
|
|
|
|
### WIP |
|
|
|
|
|
- Upcoming models: cleaned FP16 release (uniform fp16 with fp32 LayerNorms), compressed variants (W8A8, W4A16, mixed experts), 2:4 sparse checkpoints. |
|
|
- Evals: MMLU, HellaSwag, TruthfulQA, GSM8K, BBH, MT‑Bench; plus latency/throughput and memory footprint on 3090/A100. |
|
|
- Extras: scripted upload tooling, detailed model cards, and reproducible Docker workflows. |
|
|
|
|
|
|
