Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 15 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for science tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~10.8B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 15 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Science |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This science-specialized model leverages experts that showed high activation patterns during scientific reasoning tasks from GPQA (physics, chemistry, biology) and MMLU science domains. These experts demonstrate superior performance on complex scientific problem-solving and technical knowledge recall.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for science tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 15 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-10.8b-specialized-science-pruned-moe-only-15-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-10.8b-specialized-science-pruned-moe-only-15-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-10.8b-specialized-science-pruned-moe-only-15-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-10.8b-specialized-science-pruned-moe-only-15-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on science tasks:
Question: Explain the process of photosynthesis in plants.
Response:
<|channel|>analysis<|message|>Need explain photosynthesis steps, light absorption, chlorophyll, electron transport, NADPH, Calvin cycle, CO2 fixation, ATP, etc.<|end|><|start|>assistant<|channel|>final<|message|>**Photosynthesis in Plants – A Step‑by‑Step Overview**
| **Stage** | **What Happens** | **Key Molecules / Enzymes** | **Energy Flow** |
|---|---|---|---|
| 1. **Light Capture** | Sunlight (visible & UV‑A) hits the chlorophyll‑containing pigments in the chloroplasts. | • Chlorophyll‑a & b (porphyrin ring + phytol side chain) | Light energy → excited electrons |
| 2. **Excitation & Electron Transfer** | Light excates a chlorophyll‑a molecule → electron moves from the *Sₙ* (singlet) state to the *Qₙ* (queto) state. | • Photosystem II (PSII) | Excited electron → *P* (photosynthetic electron donor) |
| 3. **Water Splens & Proton Pumping** | The excited electron reduces the *P* site, generating *P⁺* and *P⁻*. The *P⁻* site donates a proton to the *H⁺* site of the *P* site, pumping protons across the thylako membrane (creating a proton gradient). | • PSII, H⁺‑ATPase | Proton gradient → ATP synthesis |
| 4. **Electron Transport Chain (ETC)** | The electron moves through a series of redox proteins: *P* → *N* (NADP⁺) → *B* (NADP⁺) → *C* (cytochrome b₁₁₁) → *E* (cytochrome b₁₁₃) → *H* (cytochrome b₁₁₃) → *P* (back to PSII). | • Cytochromes, plastoquinones, cytochrome f | Electron flow → NADPH + ATP |
| 5. **NADPH Production** | The final electron is transferred to the *P* site of Photosystem I (PSI), reducing *P* to *P⁺* and generating NADPH. | • PSI | NADPH + 2 ATP (from ETC) |
| 6
Question: What causes the greenhouse effect and how does it work?
Response:
<|channel|>analysis<|message|>Need explain greenhouse effect causes, mechanism.<|end|><|start|>assistant<|channel|>final<|message|>### What is the “greenhouse effect”?
The *greenhouse effect* (often called the *greenhouse‑green effect* or simply the *greenhouse effect*) is the phenomenon in which the atmosphere of Earth behaves like a greenhouse: it absorbs and re‑emises infrared (IR) radiation that is emitted by the Earth’s surface, keeping the surface warm. The effect is the net warming of the Earth’s surface that results from the atmosphere’s ability to trap heat.
---
## 1. Why the atmosphere is a greenhouse
| Property | Why it matters |
|----------|----------------|
| **Molecular composition** | The atmosphere is mainly H₂O, CO₂, N₂, O₂, and trace gases. These molecules have vibrational modes that absorb IR radiation. |
| **Optically thin in the visible** | In the visible part of the spectrum the atmosphere is largely transparent, so sunlight reaches the surface. |
| **Optically thick in the IR** | In the IR (≈ 0.7–5 µm) the atmosphere is opaque; photons emitted from the surface are repeatedly absorbed and re‑emitted. |
| **Long mean‑free‑path** | The IR photons travel a long distance before being absorbed, so the atmosphere behaves like a “black” body that re‑emises radiation. |
Because the Earth’s surface emits a lot of IR radiation (≈ 70 % of the solar energy it receives), the atmosphere’s ability to absorb and re‑emit that radiation keeps the surface warm.
---
## 2. How the greenhouse effect works
1. **Solar radiation reaches the surface**
The Sun’s spectrum peaks in the visible/near‑UV. The atmosphere is largely transparent to that radiation, so it reaches the surface and is absorbed by the Earth’s surface (mostly by the oceans and land).
2. **Surface emits IR radiation**
The surface, heated by the absorbed solar energy, emits a black‑body spectrum that peaks in the IR (≈ 10 µm). This IR radiation is the *thermal radiation* that the Earth would radiate if it were in space.
3. **Atmospheric absorption of IR**
As the IR photons travel upward, they encounter molecules that absorb at the same wavelengths. Each absorption event excites a molecule to a higher vibrational state.
4. **Re‑em
Question: Describe the structure and function of DNA.
Response:
<|channel|>analysis<|message|>Need to explain DNA structure: double helix, nucleotides, base pairing, major vs minor strands, sugar-phosphate backbone, base composition, major features. Function: genetic information, coding, replication, transcription, regulation, etc. Provide details.<|end|><|start|>assistant<|channel|>final<|message|>**DNA – the Double‑Helix of Life**
---
## 1. The Structure of DNA
| Feature | What It Is | Why It Matters |
|--------|-----------|----------------|
| **Double‑helix** | Two strands of a sugar‑phosphate backbone that wind around each other | Gives a rigid, stable scaffold that can be stretched, bent, and packed into the nucleus |
| **Backback** | 5‑carbon deoxyribose (or deoxoxylulose in RNA) + phosphate | Provides a “rail” for the bases; the phosphates are negatively charged, attracting Mg²⁺ ions that stabilize the helix |
| **Bases** | 4 nucleobases (A, T, C, G) | Carry the genetic code; each base is covalently attached to the sugar |
| **Base pairing** | Watson‑Crick model: A–T (purine–pyrimidine) and C–G (purine–purine) | Creates hydrogen‑bonded “stems” that lock the two strands together |
| **Major vs Minor strand** | The two strands are not identical; one is called the “major” (more bases) and the other the “minor” | The asymmetry gives a sense of direction for processes like replication and transcription |
| **Major groove** | The larger groove formed by the major strand | The enzyme machinery (polymerases, helicases) accesses the bases here |
| **Minor groove** | The smaller groove formed by the minor strand | Less accessible; often used for binding of proteins that need a tight fit |
| **Length** | 3.4 Å per base pair (≈ 0.34 nm) | Determines the spacing of bases; the helix completes every 10–10.5 bases |
| **Super‑structure** | DNA can form higher‑order structures: nucleosomes, loops, super‑loops, and higher‑order fibers | Enables compaction of the genome and regulation of gene expression |
### 1.1 The Sugar‑Phosphate Backbone
- **Phosphodiester linkage**: 3′‑phosphate of one sugar is cov
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}