Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 16 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for all tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~11.4B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 16 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | All |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This general-purpose model maintains broad capabilities across all domains while significantly reducing computational requirements. It preserves the essential routing patterns discovered across our comprehensive analysis of diverse evaluation benchmarks including GPQA, MMLU, SORRY-Bench, and Tulu3 datasets.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for all tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 16 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-11.4b-specialized-all-pruned-moe-only-16-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-11.4b-specialized-all-pruned-moe-only-16-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-11.4b-specialized-all-pruned-moe-only-16-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-11.4b-specialized-all-pruned-moe-only-16-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on all tasks:
Question: What is artificial intelligence and how does it work?
Response:
<|channel|>analysis<|message|>Need explain AI, definition, components, how it works.<|end|><|start|>assistant<|channel|>final<|message|>## 1. What is Artificial Intelligence?
Artificial Intelligence (AI) is a field of computer science that seeks to build machines—whether software, robots, or other devices—that can perform tasks that were once thought to be the exclusive domain of humans. The “intelligence” part comes from the ability to:
| **Observe** – gather data from sensors, text, images, etc. |
| **Reason** – process that data, find patterns, make inferences. |
| **Act** – produce an output (a decision, a movement, a recommendation). |
In practice, AI is a collection of algorithms, models, and hardware that let a system learn from experience, adapt to new situations, and solve problems that are too complex for simple rule‑based programming.
---
## 2. Core Ingredients of AI
| **Data** | The raw input that the system receives (images, text, sensor readings, etc.). |
| **Representation** | A way to encode that data so a machine can manipulate it (e.g., a vector, a graph, a neural‑network layer). |
| **Processing Engine** | The algorithm or model that transforms the representation into useful information (e.g., a classifier, a planner). |
| **Evaluation / Feedback** | A metric or signal that tells the system whether it did well (e.g., a score, a human rating). |
| **Learning Mechanism** | A method that updates the processing engine based on feedback (e.g., gradient‑based learning, reinforcement learning). |
---
## 3. How AI Works – A Step‑by‑Step View
1. **Input → Representation**
*Example:* A camera captures a photo → the pixels are fed into a convolutional neural network (CNN) that turns them into a 3‑dimensional “feature vector.”
2. **Representation → Processing**
*Example:* The CNN passes the feature vector through a series of layers that compute a probability distribution over possible object classes.
3. **Processing → Output**
*Example:* The system outputs “cat” with 95 % confidence.
4. **Output → Evaluation**
*Example:* A human judge says the label is correct → the system receives a positive reward.
5. **Evaluation → Learning**
*Example:* The reward is used to adjust the weights of the CNN
Question: Explain Einstein's theory of relativity in simple terms.
Response:
<|channel|>analysis<|message|>Need simple explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Einstein’s Theory of Relativity – in plain words**
---
### 1. What Einstein was trying to solve
When people were studying how objects move, they had two big puzzles:
1. **Speed of light vs. speed of a moving object** – If a light beam is shot from a moving car, does it travel faster than a beam shot from a stationary car?
2. **Time vs. distance** – If you travel very fast (close to the speed of light), does time “tick” differently for you than for someone who stays on Earth?
Einstein’s work answered both questions.
---
### 2. Two parts of the theory
| **Part** | **What it says** | **Why it matters** |
|---|---|---|
| **Galilean (Classical) Relativity** | The laws of motion (e.g., “if you push a ball, it moves”) are the same everywhere, no matter where you are. | This is the old idea that physics is the same on Earth, on a moon, or in a spaceship. |
| **General Relativity** | The universe is a 4‑dimensional space: 3 dimensions of space + 1 of time. The way objects move and how time flows are linked to the *shape* of this space. | It explains gravity, black holes, and why distant stars look slightly “off” (gravitational lensing). |
---
### 3. The key ideas in a nutshell
1. **Time is not a fixed clock**
* In Einstein’s view, time is a *coordinate* in the same way that distance is.
* If you travel very fast (close to the speed of light), the “distance” between two points in space changes, and so the *time* between two events changes too.
* That’s why astronauts on a very fast spacecraft would age a fraction of a second less than their Earth‑bound friends.
2. **Space is not a flat sheet**
* Imagine the universe as a flexible sheet that can bend.
* Gravity is the bending of this sheet: a massive object (like a planet or a black hole) pulls the sheet toward it, making the “distance” between points smaller.
* This bending changes how light travels, which is why we see the same star in slightly different places when we look from different angles.
Question: How does democracy function in modern governments?
Response:
<|channel|>analysis<|message|>Need to explain democratic functions, institutions, processes, checks, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1. What “democracy” means today
In contemporary politics “democracy” is not a single recipe but a set of **principles** that shape how power is exercised:
| Principle | Typical Institutional/Procedural Expression |
|---|---|
| **Popular sovereignty** | The people (or their elected representatives) hold ultimate authority. |
| **Rule of law** | Laws are made, interpreted, and enforced by independent institutions. |
| **Separation of powers** | Executive, legislative, and judicial branches are distinct and can check one other. |
| **Pluralism & competition** | Multiple political parties, civil‑society groups, and media outlets compete for influence. |
| **Transparency & accountability** | Public information, audits, and oversight mechanisms keep officials answerable. |
| **Participation** | Elections, referenda, public consultations, and civic engagement. |
| **Protection of rights** | Constitutional guarantees, independent courts, and human‑rights institutions. |
These principles are woven into the **constitutional architecture** of a state, and they are reinforced by **political culture** (norms, traditions, civic education) and by **civil‑society institutions** (NGOs, media, academia).
---
### 2. Core democratic institutions in modern governments
| Institution | Core Functions | Typical Checks & Balances |
|---|---|---|
| **Constitution** | Codifies the distribution of power, rights, and procedures. | Judicial review, constitutional courts. |
| **Parliament / Legislature** | Debates, drafts, and passes laws; represents the electorate. | Parliamentary oversight of the executive, committees, and public hearings. |
| **Executive (President / Prime‑Minister)** | Implements laws, runs the state, and often has veto or emergency powers. | Parliamentary approval of budgets, impeachment procedures, judicial review of executive acts. |
| **Judiciary** | Interprets laws, adjudises disputes, protects rights. | Independent appointment processes, judicial oversight, appellate review. |
| **Electoral System** | Conducts free, fair, and regular elections. | Independent electoral commissions, international observers, legal remedies for electoral disputes. |
| **Civil‑Society & Media** | Provide information, critique, and mobilize citizens. | Freedom of expression, press‑law protections, watchdog NGOs. |
| **Public‑Sector
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}