Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 13 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for all tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~9.6B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 13 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | All |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This general-purpose model maintains broad capabilities across all domains while significantly reducing computational requirements. It preserves the essential routing patterns discovered across our comprehensive analysis of diverse evaluation benchmarks including GPQA, MMLU, SORRY-Bench, and Tulu3 datasets.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for all tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 13 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-9.6b-specialized-all-pruned-moe-only-13-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-9.6b-specialized-all-pruned-moe-only-13-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-9.6b-specialized-all-pruned-moe-only-13-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-9.6b-specialized-all-pruned-moe-only-13-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on all tasks:
Question: What is artificial intelligence and how does it work?
Response:
<|channel|>analysis<|message|>Need explain AI definition, types, how it works: data, models, training, inference, neural networks, etc.<|end|><|start|>assistant<|channel|>final<|message|>## 1. What is Artificial Intelligence?
Artificial Intelligence (AI) is a set of computational techniques that let machines *think*—that is, that they can observe data, learn from it, and then use that knowledge to solve problems, make predictions, or generate new content. In short, AI is a **machine‑learning system** that can:
| Feature | Example |
|--------|--------|
| **Pattern recognition** | Spoting a face in a photo, a stock price trend, or a word in a sentence |
| **Generalization** | Infitting a rule that applies to many unseen examples |
| **Reasoning** | Choosing the best answer to a question, or the best route to a destination |
| **Creativity** | Writing a poem, composing a melody, or designing a new product |
AI is *not* a single algorithm; it is a family of methods that share the same goal: **to emulate human intelligence** in a purely digital medium.
---
## 2. How AI Works – The Core Ingredients
| Step | What Happens | Why It Matters |
|------|--------------|----------------|
| **1. Data Collection** | Gather a large set of observations (images, text, sensor readings, etc.) | Provides the “experience” the machine needs to learn from |
| **2. Representation** | Convert data into a machine‑friendly format (e.g., tensors, graphs, or embeddings) | Makes the data machine‑readable and comparable |
| **3. Model Building** | Build a hypothesis‑based structure (a *model*) that can explain the patterns in the data | Gives the machine a *theory* to test |
| **4. Training** | Feed the model with data and let it adjust its internal parameters (weights) so it best predicts unseen data | The model learns from the data, just like a human learning from experience |
| **5. Validation** | Test the model on a new dataset that the machine never saw before | Checks if the model truly generalized, not just memorized |
| **6. Deployment** | Use the trained model to answer new questions or solve tasks in real time | The machine now *acts* intelligently |
### 2.1 The Building Blocks
| Block | What Is It | How It Works |
|-------|------------|
Question: Explain Einstein's theory of relativity in simple terms.
Response:
<|channel|>analysis<|message|>Need simple explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Einstein’s Theory of Relativity – in plain words**
---
### 1. What the theory says
- **Space and time are not fixed.**
In the universe, the distance between two points and the speed of a moving object can change depending on the *direction* in which they are measured.
- **The same thing can look different from different viewpoints.**
If you look at a particle from one side of the universe, it may seem to move slowly. If you look from another side, it may seem to move fast. The “speed” you measure depends on the *angle* you’re looking from.
- **The universe is a “relativ” (relative) playground.**
The laws that we use on Earth (like “if two objects collide, they exchange energy”) still hold, but the *amount* of energy and the *speed* of objects can shift when you change the viewpoint.
---
### 2. How it works – a quick analogy
Imagine a **two‑dimensional world** (like a flat sheet of paper).
- **Horizontal axis (x)**: Think of this as “time.”
- **Vertical axis (y)**: Think of this as “space.”
If you draw a straight line from the top to the bottom of the sheet, you’re looking along the *vertical* direction.
If you draw a straight line from the left to the right, you’re looking along the *horizontal* direction.
Now, suppose you tilt the sheet at a 45° angle.
- The line that was straight vertically is now *tilted*; it’s no longer a straight line in the sheet’s geometry.
- The same line that was straight horizontally is also tilted.
In relativity, the *tilting* of the sheet is like the *direction* in space. The “straightness” (or “straightness” of a line) is like the *speed* of an object. When you tilt the sheet, the straightness changes – just like how the speed of an object changes when you look from a different direction.
---
### 3. Key ideas
| Concept | Everyday picture | Relativity twist |
|---------|-------------------|-------------------|
| **Speed** | A car moving at 60 km/h | The car’s speed can be 60 km/h *or* 120 km/h depending on the angle
Question: How does democracy function in modern governments?
Response:
<|channel|>analysis<|message|>Need explain concept, institutions, checks, participation, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1. What is “democracy” in a modern state?
In practice, a democracy is a system in which **the people have the ultimate authority** over the state’s laws, policies, and leaders. The authority is exercised through a set of institutions and procedures that are designed to be:
| **Key Feature** | **How it works in practice** | **Why it matters** |
|-------------------|---------------------------|---------------------|
| **Representative elections** | Citizens vote for elected officials (party‑based or independent). | Gives a clear mandate for the government. |
| **Rule of the law** | Laws are made, interpreted, and enforced by independent bodies. | Protects rights and limits power. |
| **Checks & balances** | Separation of powers (executive, legislative, judiciary). | Prevents concentration of power. |
| **Freedom of expression & assembly** | Citizens can speak, write, protest, and organize. | Enables scrutiny and accountability. |
| **Transparency & accountability** | Public records, audits, media scrutiny. | Keeps officials answerable to the public. |
| **Pluralism & minority rights** | Legal guarantees for minority groups, plural parties. | Ensures no single group dominates. |
These are the “building blocks” that modern democries use to keep the system functioning.
---
### 2. How the system actually operates
#### 2.1 Representative institutions
1. **Parliament / Congress**
*Elected members debate, draft laws, and vote on policy.*
*They are the “voice” of the electorate, but they are also subject to scrutiny by the press and civil‑rights groups.*
2. **Executive (President, Prime‑Minister, etc.)**
*Runs day‑to‑day policy, implements laws, and represents the state.*
*Must be elected (or appointed by parliament) and is accountable through elections, parliamentary oversight, and the media.*
3. **Judiciary**
*Interinterpres laws, adjudises disputes, and checks executive power.*
*Its independence is protected by separate appointment processes and legal safeguards.*
#### 2.2 Electoral mechanisms
| **Type** | **What it does** | **Typical safeguards** |
|----------|-----------------|------------------------|
| **Ball‑ball** | Simple, quick, often used in local
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}