Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 11 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for all tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~8.4B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 11 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | All |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This general-purpose model maintains broad capabilities across all domains while significantly reducing computational requirements. It preserves the essential routing patterns discovered across our comprehensive analysis of diverse evaluation benchmarks including GPQA, MMLU, SORRY-Bench, and Tulu3 datasets.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for all tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 11 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-8.4b-specialized-all-pruned-moe-only-11-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-8.4b-specialized-all-pruned-moe-only-11-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-8.4b-specialized-all-pruned-moe-only-11-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-8.4b-specialized-all-pruned-moe-only-11-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on all tasks:
Question: What is artificial intelligence and how does it work?
Response:
<|channel|>analysis<|message|>Need explain AI, definition, types, how it works: data, models, training, inference, neural networks, etc.<|end|><|start|>assistant<|channel|>final<|message|>## What Is Artificial Intelligence?
Artificial Intelligence (AI) is a set of computational systems that can **undercome** or **underperform** tasks that were traditionally performed by humans. In other words, AI is a machine‑learning system that can:
* **Observe** patterns in data (e.g., text, images, numbers, sounds).
* **Infer** relationships or predictions from those patterns.
* **Act** by generating new outputs (answers, predictions, actions) that are useful to humans.
AI is not a single algorithm or a single “brain”; it is a **collection of techniques** that can be combined to produce a system that behaves like a human mind in a specific domain.
---
## Core Ideas Behind AI
| Concept | What it does | How it works |
|--------|----------------|----------------|
| **Data** | Stores a set of facts (e.g., a corpus of text, a set of images, a set of numbers). | Usually in a **memory** (e.g., a database, a tensor‑array). |
| **Model** | A mathematical representation of the data that captures the relationships between data points. | Often a **probability‑based** or **functional‑based** model (e.g., a neural network, a decision‑tree). |
| **Training** | The process of learning the model from the data. | Reiterative: repeatedly expose the model to data and reward it for correct predictions. |
| **Inference** | Using the trained model to answer a new question or perform a new task. | Apply the model to new data and output the prediction. |
| **Evaluation** | Measure how well the model performs. | Compare predictions to ground truth or to human performance. |
---
## How AI Works in Practice
### 1. **Data Collection**
* **Input**: The system receives a set of inputs (text, images, numbers, audio, etc.).
* **Storage**: The inputs are stored in a memory (e.g., a database, a tensor array).
### 2. **Model Construction**
* **Mathematical Representation**: The data is represented by a **model**.
* **Neural Networks** (e.g., convolutional networks, recurrent‑state networks) for images/text.
* **Decision Trees**
Question: Explain Einstein's theory of relativity in simple terms.
Response:
<|channel|>analysis<|message|>Need simple explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Einstein’s theory of relativity** is a way of understanding how the universe works when we look at the motion of objects that are far apart from each point in space.
---
### 1. What we know before
* **Newton’s law of gravity** (the *law of inertia*):
If one object is already moving in a direction, any other object will move in the opposite direction, and the two objects will keep a constant speed relative to each other.
In other words, if a ball is moving to the right, another ball will move to the left, and the two balls will keep the same speed.
* **The *law of simultity*** (the *law of simultity*):
If two points are at the same distance from each other, the two points will see the same motion at the same time.
In other words, if two points are at the same distance, the two points will see the same motion at the same time.
These two laws are true when the points are **close** to each other (e.g., in a laboratory or on a planet). But they break when the points are **far** from each other (e.g., between a planet and a star).
---
### 2. What Einstein noticed
When we look at a *very far* distance, the two points do not see the same motion at the same time.
Instead, the motion that we see at one point is *different* from the motion we see at the other point.
This is because the two points are far enough that the *time* between the two points is *not the same*.
---
### 3. The new idea
Einstein’s theory says that:
* **The *law of simultity* is wrong for far distances.**
The two points do not see the same motion at the same time.
The motion we see at one point is *different* from the motion we see at the other point.
* **The *law of inertia* still works.**
Even when the points are far apart, if one point is moving in a direction, the other point will move in the opposite direction, and the two points will keep a constant speed relative to each other.
---
### 4. How it works
Imagine a *two‑way* universe:
* **Space** – the *distance* between two points.
Question: How does democracy function in modern governments?
Response:
<|channel|>analysis<|message|>Need explain role, institutions, checks, participation, checks, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1. The Core Idea of Democracy in Modern Governments
At the most level, democracy is the **“rule by the people”** principle. In modern states it is expressed by a set of mechanisms that let the citizenry shape the laws, the leaders, and the policies that govern them. These mechanisms are not a single thing; they are a **network of institutions, norms, and practices** that together keep the government accountable, responsive, and legitimate.
---
## 2. How Democracy Functions in Practice
| **Institution / Mechanism** | **What It Does** | **Typical Features** | **Why It Matters** |
|---|---|---|---|
| **Representative Elections** | Citizens choose leaders (political parties, candidates) who will run the government. | • Regular (e.g., 4‑year, 2‑year) elections<br>• Ball ballots, polling, campaign rules<br>• Electoral commissions, independent oversight | • Gives a mandate for policy decisions<br>• Creates a “check‑on” for leaders |
| **Parleg‑al Committees / Councors** | Elected representatives debate, draft, and vote on laws. | • Legislative chambers (parpar, parliament, senate) <br>• Commitors, caucies, committees<br>• Majority or majority‑plus‑minority rules | • Transensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensensens
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}