Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 13 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for science tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~9.6B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 13 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Science |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This science-specialized model leverages experts that showed high activation patterns during scientific reasoning tasks from GPQA (physics, chemistry, biology) and MMLU science domains. These experts demonstrate superior performance on complex scientific problem-solving and technical knowledge recall.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for science tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 13 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-9.6b-specialized-science-pruned-moe-only-13-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-9.6b-specialized-science-pruned-moe-only-13-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-9.6b-specialized-science-pruned-moe-only-13-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-9.6b-specialized-science-pruned-moe-only-13-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on science tasks:
Question: Explain the process of photosynthesis in plants.
Response:
<|channel|>analysis<|message|>Need to explain steps, light absorption, electron transfer, Calvin cycle, etc.<|end|><|start|>assistant<|channel|>final<|message|>**Photosynthesis – the “green” chemistry of life**
Plants turn sunlight into chemical energy to grow, reproduce, and maintain their ecosystems. The process is a coordinated dance of light‑energy conversion, electron transfer, and carbon‑fixing chemistry that takes only a few seconds in a leaf but is essential for the biosphere.
---
## 1. Light‑energy capture: the chlorophyll antenna
| **Step** | **What happens** | **Why important** |
|---|---|---|---|
| **Photon absorption** | Chlorophyll‑a and chlorophyll‑b (plus accessory pigments) absorb photons in the blue‑green part of the visible spectrum (≈400–650 nm). | These pigments act as “antennae” that can harvest the energy of photons. |
| **Excite formation** | The absorbed photon promotes an electron from the *ground state* (S₀) to an *excited state* (S₁) of the chlorophyll molecule. | The excited state is short‑lifetime (≈10⁻¹�⁴ s) and must be converted to a usable chemical form. |
| **Charge‑separation** | The excited chlorophyll rapidly undergoes *charge‑separation* (a “photochemical” step). An electron is promoted to the *conhole* (S₁ → S₁) and a hole (S₁ → S₁) is left behind. | This creates a “hot” electron that can be used for a chemical reaction. |
---
## 2. Electron transfer to the *electron‑transport chain*
| **Step** | **What happens** | **Key enzymes** |
|---|---|---|---|
| **Electron transfer** | The hot electron is passed to the *electron‑transport chain* (ETC) embedded in the thromosome (a protein complex). | The ETC is a series of redox reactions that ultimately produce a proton gradient and ATP. |
| **ATP synthesis** | The ETC pumps protons across the *inner* and *outer* membranes of the mitochondoid‑like thromosome, creating a proton motive force (ΔpH ≈ 0.5 mV). | The proton gradient is used to drive the *hydrolysis* of ATP‑generating reactions. |
| **
Question: What causes the greenhouse effect and how does it work?
Response:
<|channel|>analysis<|message|>Need explain greenhouse effect causes, greenhouse gases, absorption, re-emission, warming.<|end|><|start|>assistant<|channel|>final<|message|>### What is the “greenhouse effect” (or greenhouse effect)?
The greenhouse effect is the warming of the Earth’s surface and atmosphere caused by the trapping of heat that is emitted from the Earth’s surface and from the Sun. It is the same as the greenhouse effect in a laboratory: a container of a gas that absorbs and re‑emits infrared radiation, keeping the temperature higher than it would be if the gas were not present.
---
## 1. What gases are responsible?
The main greenhouse gases (GHGs) are:
| Gas | Source | Effect on the atmosphere |
|------|--------|---------------------------|
| **CO₂** (carbon dioxide) | Fossile‑fuel combustion, methane oxidation, volcanic CO₂ | Strong absorber of IR, 1‑2 % of the Earth’s energy budget |
| **CH₄₄** (methane) | Natural gas, bi‑methane, fossil‑fuel combustion | Strong IR absorber, 1‑2 % of the energy budget |
| **N₂O₄** (nitrous oxide) | Fossile‑fuel combustion, industrial processes | Strong IR absorber, 0.5 % of the energy budget |
| **H₂O** (water vapor) | Evaporised water from the surface | Very strong IR absorber, but it is a natural feedback |
| **C₄H₁₀** (C‑clane) | Fossile‑fuel combustion, industrial processes | Strong IR absorber, 0.5 % of the energy budget |
| **C₂O₂** (CO₂) | Fossile‑fuel combustion, methane oxidation | Strong IR absorber, 1‑2 % of the energy budget |
The other gases (e.g., N₂, O₂, Ar, He) are essentially transparent to IR radiation and do not contribute to the greenhouse effect.
---
## 2. How does the greenhouse effect work?
1. **Solar radiation reaches the Earth**
The Sun emits a broad spectrum of radiation. About 30 % of the solar energy is absorbed by the Earth’s surface and atmosphere. The rest is reflected or transmitted into space.
2. **The Earth emits thermal radiation**
The Earth is in thermal equilibrium: the absorbed solar energy is re‑emitted as a black‑body spectrum (mostly in
Question: Describe the structure and function of DNA.
Response:
<|channel|>analysis<|message|>Need to explain DNA structure: double helix, base pairs, sugar, phosphate, nitrogen, etc. Function: genetic code, replication, transcription, translation, regulation, etc.<|end|><|start|>assistant<|channel|>final<|message|>**DNA: the double‑helix of life**
---
## 1. The “structure” of DNA
| **Component** | **What it does** | **Key features** |
|----------------|----------------|----------------|
| **Double helix** | Two strands wind around each way | Right‑handed (R) and left‑handed (L) helices |
| **Base pairs** | 10 pairs per 3.4 nm (≈ 10 nm per 100 bp) | 10 bp = 3.4 nm; 10 bp = 10 nm (≈ 10 nm per 100 bp) |
| **Sugar (deph‑phosph‑deph‑phosph)** | Provides the backbone | 5 × 10⁻¹�⁰ kJ/mol of bond energy |
| **Phosphorus (P)** | Provides the backbone | 5 × 10⁻¹��⁰ kJ/mol of bond energy |
| **Nitrogen (N)** | Provides the backbone | 5 × 10⁻¹��⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹�⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹�⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹�⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹�⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹⁰ kJ/mol of bond energy |
| **Hydrogen (H)** | Provides the backbone | 5 × 10⁻¹⁰ kJ/mol
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}