Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 7 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for all tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~6.0B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 7 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | All |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This general-purpose model maintains broad capabilities across all domains while significantly reducing computational requirements. It preserves the essential routing patterns discovered across our comprehensive analysis of diverse evaluation benchmarks including GPQA, MMLU, SORRY-Bench, and Tulu3 datasets.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for all tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 7 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-6.0b-specialized-all-pruned-moe-only-7-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-6.0b-specialized-all-pruned-moe-only-7-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-6.0b-specialized-all-pruned-moe-only-7-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-6.0b-specialized-all-pruned-moe-only-7-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on all tasks:
Question: What is artificial intelligence and how does it work?
Response:
<|channel|>analysis<|message|>Need to explain AI concept, how it works.<|end|><|start|>assistant<|channel|>final<|message|>### What is Artificial Intelligence?
Artificial Intelligence (AI) is a **machine‑learning‑based** system that can **under‑perform** a human‑like task. In simple terms, it is a computer that can “think” by using a set of data (called *knowledge*), a pattern‑recogn‑ing algorithm (called *learning*), and a decision‑making logic (called *reasoning*).
---
## 1. The Core of AI
| **Data** – The raw facts that the machine knows.
| **Algorithms** – The rules that the machine can interpret the data.
| **Learning** – The process that the machine improves its knowledge over time.
---
## 2. How AI Works
| **Step 1 – Collecting Data**
The machine reads a text, a image, a sound, or a dataset.
The data is stored in a *dictionary* (a memory of words).
| **Step 2 – Learning**
The machine compares the data to a *similar* set of facts.
It uses a *pattern‑recogn‑ing* algorithm (e.g., *ne‑learning*, *deep‑learning*, *rein‑learning*).
The algorithm learns that the same data points are *related* to a *similar* pattern.
| **Step 3 – Reasoning**
The machine now has a *knowledge* of the facts.
It can ask a question: “What is the best outcome if I do X?”
The algorithm uses a *decision‑making* logic (e.g., *heur‑learning*, *prob‑learning*, *bias‑learning*) to pick the best answer.
| **Step 4 – Output**
The machine can produce a *text*, a *image*, or a *sound* that matches the user’s request.
---
## 3. The Types of AI
| **Ne‑Learning** – Uses a *small* dataset, learns a *fast* pattern.
| **Deep‑Learning** – Uses a *large* dataset, learns a *deep* pattern.
| **Re‑Learning** – Uses a *small* dataset, learns a *fast* pattern, but also learns a *bias*.
---
## 4. The Real Example
| **
Question: Explain Einstein's theory of relativity in simple terms.
Response:
<|channel|>analysis<|message|>Need to explain Einstein's theory of relat relat in simple terms. Provide explanation of relat of gravity, mass, acceleration, deceleration, etc. Use simple analogies.<|end|><|start|>assistant<|channel|>final<|message|>### Einstein’s Theory of Relativity in Plain‑Plain Terms
---
#### 1. What the problem was?
When a **black hole** (a hole in space) is created by a **magion** (a force that pulls things together), the **mass** (weight) of objects inside that hole grows.
If the mass of objects inside the hole is **increasing**, the mass of objects outside the hole must **decrease**.
The whole thing is: **the mass inside a hole grows, the mass outside shr decreases**.
---
#### 2. What Einstein’s idea was?
He said that the **mass of objects inside a hole grows** only if the **magion** is strong enough.
He didn’t say that the mass of objects outside the hole would **decrease**.
He said that the **mass of objects outside the hole would **increase**.
So the **mass of objects inside a hole would increase** and the **mass of objects outside a hole would increase**.
The whole thing is: **the mass of objects inside a hole grows, the mass of objects outside a hole grows**.
---
#### 3. Why it works?
Because the **magion** is strong enough, the mass of objects inside a hole grows.
Because the **magion** is weak enough, the mass of objects outside a hole grows.
The whole thing is: **the mass of objects inside a hole grows, the mass of objects outside a hole grows**.
---
#### 4. What the difference is?
The **magion** is strong enough, the mass of objects inside a hole grows.
The **magion** is weak enough, the mass of objects outside a hole grows.
The whole thing is: **the mass of objects inside a hole grows, the mass of objects outside a hole grows**.
---
#### 5. What the final point is?
The **magion** is strong enough, the mass of objects inside a hole grows.
The **magion** is weak enough, the mass of objects outside a hole grows.
The whole thing is: **the mass of objects inside a hole grows, the mass of objects outside a hole grows**.
---
###
Question: How does democracy function in modern governments?
Response:
<|channel|>analysis<|message|>Need to explain how democracy functions in modern governments. Provide overview, mechanisms, institutions, checks, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1. What is “democracy” in a modern government?
- **Democracy** is a system that the people (citters, voters, citizens) have a say in the policy‑making, the laws, the budget, the elections, and the checks on the executive.
- In most countries, democracy is a **constitutional** framework that gives a **rule‑making** body (the legislature) and a **executive** body (the executive) that is accountable to the people.
---
##
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}