Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 14 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for science tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~10.2B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 14 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Science |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This science-specialized model leverages experts that showed high activation patterns during scientific reasoning tasks from GPQA (physics, chemistry, biology) and MMLU science domains. These experts demonstrate superior performance on complex scientific problem-solving and technical knowledge recall.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for science tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 14 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-10.2b-specialized-science-pruned-moe-only-14-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-10.2b-specialized-science-pruned-moe-only-14-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-10.2b-specialized-science-pruned-moe-only-14-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-10.2b-specialized-science-pruned-moe-only-14-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on science tasks:
Question: Explain the process of photosynthesis in plants.
Response:
<|channel|>analysis<|message|>Need to explain photosynthesis.<|end|><|start|>assistant<|channel|>final<|message|>**Photosynthesis – the life‑fueling process that powers plants**
---
### 1. What is photosynthesis?
Photosynthesis is the biochemical conversion of light energy into chemical energy that plants use to grow, reproduce, and maintain their internal chemistry. In short, plants “burn” sunlight to make sugars, which are the building blocks of life.
There are two main stages:
| **Stage** | **What happens** | **Key enzymes / proteins** | **Where it occurs** |
|---|---|---|---|
| **Light‑energy capture** | Sunlight (visible light) is absorbed by pigments (chlorophyll a, b, and accessory pigments). | **Chlorophyll a** (main pigment) | **Leaf cell walls** (photosynthetic cells) |
| **Chemical conversion** | Light energy is used to reduce water (H₂O) into oxygen (O₂) and carbon dioxide (CO₂). | **Photosystem II (PSII)** – reduces water → O₂ + H₂O; **Photosystem I (PSI)** – oxidizes NADP → NADPH | **Leaf chloroplasts** (inner thylako membrane) |
---
### 2. The molecular machinery
#### 2.1 The light‑absorbing complex – Photosystem II (PSII)
1. **Photon absorption** – Chlorophyll a absorbs a photon (≈680 nm).
2. **Excited state** – The excited chlorophyll a transfers an electron to the *electron‑transport chain* (a series of proteins in the thylako membrane).
3. **Water splitting** – The electron reaches the *cytochrome b₄* complex, which splits water into two protons (H⁺) and an oxygen molecule (O₂).
4. **Propm generation** – The two protons are pumped into the *stilum* (interthilako space) by the *prok H⁺‑ATPase*.
#### 2.2 The electron‑transport chain
- **Complex of proteins**: cytochrome b₄, cytochrome c₄, cytochrome f, cytochrome a₁₂₃, cytochrome a₇�₈, cytochrome b₁�₁, cytochrome b₁�₁₁,
Question: What causes the greenhouse effect and how does it work?
Response:
<|channel|>analysis<|message|>Need explain greenhouse effect causes, mechanism.<|end|><|start|>assistant<|channel|>final<|message|>### 1. What is the “greenhouse effect”?
The **greenhouse effect** (or **greenhouse gas effect**) is the process by which the Earth’s atmosphere warms because it traps heat that is emitted from the surface. The main driver is the absorption and re‑emission of infrared (IR) radiation by the gases that are present in the atmosphere.
---
## 2. Why the atmosphere is a “greenhouse”
| **Key property** | **Why it matters** |
|-------------------|---------------------|
| **Low absorption of visible light** | Solar radiation that reaches the surface is largely in the visible spectrum; the atmosphere is almost transparent to it. |
| **Strong absorption of IR** | The surface emits a black‑body spectrum that peaks in the IR. Greenhouse gases absorb this IR and re‑emit it. |
| **Long‑time‑scale re‑emission** | The re‑emitted IR is not directed away; it is trapped and eventually escapes only after the atmosphere has cooled. |
Because the atmosphere is “transparent” to the Sun but “opaque” to the Earth’s own thermal radiation, the Earth behaves like a greenhouse: it keeps a little of its heat inside.
---
## 3. The main greenhouse gases
| Gas | Abundance (ppm) | Radiative properties |
|-----|-------------------|------------------------|
| **Water vapor (H₂O)** | ~ 2 000 | Strong IR absorber; dominates short‑time weather changes. |
| **Carbon dioxide (CO₂)** | ~ 400 | Strong IR absorber; main anthropogenic driver. |
| **Methane (CH₄)** | ~ 1.8 | Strong IR absorber; increases with fossil‑fuel use. |
| **Nitrous oxide (N₂O)** | ~ 30 | Strong IR absorber; produced from agriculture. |
| **Oxygen (O₂)** | ~ 21 000 | Abs absorbs UV, but also some IR. |
| **Other trace gases** (e.g., C₂H₆, H₂S, etc.) | < 1 | Minor but non‑negligible. |
The most important are CO₂, CH₄, N₂O, and H₂O.
---
## 4. How the greenhouse effect works (radiative balance)
1. **Solar radiation enters the atmosphere
Question: Describe the structure and function of DNA.
Response:
<|channel|>analysis<|message|>Need to explain DNA structure: double heline, base pairs, major components, sugar-phosphate backbone, major bases, minor bases, major vs minor. Function: genetic information, coding, replication, transcription, regulation, etc.<|end|><|start|>assistant<|channel|>final<|message|>**DNA (Deoxyribonucleic Acid)** is the fundamental genetic material of all living organisms. It is a long, linear polymer that stores the information needed for an organism’s identity, its proteins, and the regulatory networks that govern its life. The structure and function of DNA can be understood in two complementary ways:
| **Structure** | **Function** |
|---|---|
| **Double‑helical backbone** – two strands of sugar‑phosphate (de‑ox‑ribose‑phosphate) run antiparphally, forming a right‑handed helix. The two strands are held together by 3‑5 hydrogen bonds between complementary bases. | **Stability & fidelity** – the double helix protects the genetic information from heat, chemical damage, and random mutations. The hydrogen‑bonded base pairs give a high‑order, low‑error replication system. |
| **Major (canonical) bases** – Adenine (A), Thymine (T), Cytosine (C), Guanine (G). These are the four nucleobases that pair in a Watson‑Crick fashion (A‑T, G‑C). | **Encoding of genetic code** – the sequence of A, T, C, G determines the information that will be read by the cell’s machinery. The canonical bases are the most chemically stable and are the ones that the cell’s enzymes recognize. |
| **Minor (rare) bases** – Inorganic or modified bases such as 5‑methyl‑2‑thymine (mT), 5‑hydroxymethyl‑2‑thymine (hmT), 5‑hydroxymethyl‑2‑deoxy‑thymine (hmdT), 5‑hydroxymethyl‑2‑deoxy‑thymine‑N‑methyl‑guanine (hmdT‑mG), etc. These are present at <0.1 % of the total bases. | **Regulation & adaptation** – minor bases can modulate the local structure of the helix, affect the binding of proteins, and can be involved in stress responses, epigenetic regulation, or
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}