|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: |
|
|
- openai/gpt-oss-20b |
|
|
tags: |
|
|
- mlx |
|
|
- gpt |
|
|
- openai |
|
|
- chatGPT |
|
|
--- |
|
|
|
|
|
|
|
|
# OpenAI GPT OSS 20B - MLX 4-bit |
|
|
|
|
|
## Model Description |
|
|
|
|
|
This is a 4-bit quantized version of the OpenAI GPT OSS 20B model, optimized for Apple Silicon using the MLX framework. The model was successfully converted from the original `gpt_oss` architecture to MLX format using the development version of `mlx-lm` (v0.26.3). |
|
|
|
|
|
>[!TIP] Tip: For best results with tool calling and reasoning, update your LMStudio to the latest version (0.3.22) |
|
|
|
|
|
## Quick Start |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install mlx-lm |
|
|
``` |
|
|
|
|
|
### Usage |
|
|
|
|
|
```python |
|
|
from mlx_lm import load, generate |
|
|
|
|
|
# Load the quantized model |
|
|
model, tokenizer = load("/Users/thomas/Documents/Model Weights/gpt-oss-20b-MLX-4bit") |
|
|
|
|
|
# Generate text |
|
|
prompt = "Explain quantum computing in simple terms:" |
|
|
response = generate(model, tokenizer, prompt=prompt, verbose=True, max_tokens=512) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Model Architecture |
|
|
- **Model Type**: `gpt_oss` (GPT Open Source) |
|
|
- **Architecture**: `GptOssForCausalLM` |
|
|
- **Parameters**: ~20 billion parameters |
|
|
- **Quantization**: 4-bit precision (4.504 bits per weight) |
|
|
|
|
|
### Core Parameters |
|
|
| Parameter | Value | |
|
|
|-----------|-------| |
|
|
| Hidden Size | 2,880 | |
|
|
| Intermediate Size | 2,880 | |
|
|
| Number of Layers | 24 | |
|
|
| Attention Heads | 64 | |
|
|
| Key-Value Heads | 8 | |
|
|
| Head Dimension | 64 | |
|
|
| Vocabulary Size | 201,088 | |
|
|
|
|
|
### Mixture of Experts (MoE) Configuration |
|
|
- **Number of Local Experts**: 32 |
|
|
- **Experts per Token**: 4 |
|
|
- **Router Auxiliary Loss Coefficient**: 0.9 |
|
|
- **SwiGLU Limit**: 7.0 |
|
|
|
|
|
### Attention Mechanism |
|
|
- **Attention Type**: Hybrid sliding window and full attention |
|
|
- **Sliding Window Size**: 128 tokens |
|
|
- **Max Position Embeddings**: 131,072 tokens |
|
|
- **Initial Context Length**: 4,096 tokens |
|
|
- **Attention Pattern**: Alternating sliding and full attention layers |
|
|
|
|
|
### RoPE (Rotary Position Embedding) Configuration |
|
|
- **RoPE Theta**: 150,000 |
|
|
- **RoPE Scaling Type**: YaRN (Yet another RoPE extensioN) |
|
|
- **Scaling Factor**: 32.0 |
|
|
- **Beta Fast**: 32.0 |
|
|
- **Beta Slow**: 1.0 |
|
|
|
|
|
### Quantization Details |
|
|
- **Quantization Method**: MLX 4-bit quantization |
|
|
- **Group Size**: 64 |
|
|
- **Effective Bits per Weight**: 4.504 |
|
|
- **Size Reduction**: 13GB → 11GB (~15% reduction) |
|
|
|
|
|
### File Structure |
|
|
``` |
|
|
gpt-oss-20b-MLX-4bit/ |
|
|
├── config.json # Model configuration |
|
|
├── model-00001-of-00003.safetensors # Model weights (part 1) |
|
|
├── model-00002-of-00003.safetensors # Model weights (part 2) |
|
|
├── model-00003-of-00003.safetensors # Model weights (part 3) |
|
|
├── model.safetensors.index.json # Model sharding index |
|
|
├── tokenizer.json # Tokenizer configuration |
|
|
├── tokenizer_config.json # Tokenizer settings |
|
|
├── special_tokens_map.json # Special tokens mapping |
|
|
├── generation_config.json # Generation parameters |
|
|
└── chat_template.jinja # Chat template |
|
|
``` |
|
|
|
|
|
## Performance Characteristics |
|
|
|
|
|
### Hardware Requirements |
|
|
- **Platform**: Apple Silicon (M1, M2, M3, M4 series) |
|
|
- **Memory**: ~11GB for model weights |
|
|
- **Recommended RAM**: 16GB+ for optimal performance |
|
|
|
|
|
### Layer Configuration |
|
|
The model uses an alternating attention pattern across its 24 layers: |
|
|
- **Even layers (0, 2, 4, ...)**: Sliding window attention (128 tokens) |
|
|
- **Odd layers (1, 3, 5, ...)**: Full attention |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Tokenizer |
|
|
- **Type**: Custom tokenizer with 201,088 vocabulary size |
|
|
- **Special Tokens**: |
|
|
- EOS Token ID: 200,002 |
|
|
- Pad Token ID: 199,999 |
|
|
|
|
|
### Model Configuration |
|
|
- **Hidden Activation**: SiLU (Swish) |
|
|
- **Normalization**: RMSNorm (ε = 1e-05) |
|
|
- **Initializer Range**: 0.02 |
|
|
- **Attention Dropout**: 0.0 |
|
|
- **Attention Bias**: Enabled |
|
|
|
|
|
## Conversion Process |
|
|
|
|
|
This model was successfully converted using: |
|
|
- **MLX-LM Version**: 0.26.3 (development branch) |
|
|
- **Conversion Command**: |
|
|
```bash |
|
|
python3 -m mlx_lm convert \ |
|
|
--hf-path "/path/to/openai-gpt-oss-20b" \ |
|
|
--mlx-path "/path/to/gpt-oss-20b-MLX-4bit" \ |
|
|
--quantize \ |
|
|
--q-bits 4 |
|
|
``` |
|
|
|
|
|
## Known Limitations |
|
|
|
|
|
1. **Architecture Specificity**: This model uses the `gpt_oss` architecture which is only supported in MLX-LM v0.26.3+ |
|
|
2. **Platform Dependency**: Optimized specifically for Apple Silicon; may not run on other platforms |
|
|
3. **Quantization Trade-offs**: 4-bit quantization may result in slight quality degradation compared to full precision |
|
|
|
|
|
## Compatibility |
|
|
|
|
|
- **MLX-LM**: Requires v0.26.3 or later for `gpt_oss` support |
|
|
- **Apple Silicon**: M1, M2, M3, M4 series processors |
|
|
- **macOS**: Compatible with recent macOS versions supporting MLX |
|
|
|
|
|
## License |
|
|
|
|
|
Please refer to the original OpenAI GPT OSS 20B model license terms. |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
- Original model by OpenAI |
|
|
- MLX framework by Apple Machine Learning Research |
|
|
- Quantization achieved using `mlx-lm` development tools |
|
|
|
|
|
--- |
|
|
|
|
|
**Model Size**: 11GB |
|
|
**Quantization**: 4-bit (4.504 bits/weight) |
|
|
**Created**: August 6, 2025 |
|
|
**MLX-LM Version**: 0.26.3 (development) |
