Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 22 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for science tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~14.9B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 22 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Science |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This science-specialized model leverages experts that showed high activation patterns during scientific reasoning tasks from GPQA (physics, chemistry, biology) and MMLU science domains. These experts demonstrate superior performance on complex scientific problem-solving and technical knowledge recall.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for science tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 22 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-14.9b-specialized-science-pruned-moe-only-22-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-14.9b-specialized-science-pruned-moe-only-22-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-14.9b-specialized-science-pruned-moe-only-22-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-14.9b-specialized-science-pruned-moe-only-22-experts")
# Generate with the model
messages = [
{"role": "user", "content": "Explain the process of photosynthesis in plants."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on science tasks:
Question: Explain the process of photosynthesis in plants.
Response:
<|channel|>analysis<|message|>Need concise explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Photosynthesis in Plants – A Step‑by‑Step Overview**
| Step | What Happens | Key Players | Result |
|------|--------------|-------------|--------|
| 1. **Light Capture** | Sunlight hits the leaf. | Chlorophyll‑a & b (pigments in chloroplasts) | Energy is absorbed and converted to chemical energy. |
| 2. **Water Splitting (Photolysis)** | Light energy splits H₂O into H⁺, e⁻, and O₂. | Photosystem II (PSII) + water‑splitting complex | O₂ released to atmosphere; electrons start the electron‑transport chain. |
| 3. **Electron Transport & ATP Formation** | Electrons travel through PSII → plastoquinone → cytochrome b₆f → Photosystem I (PSI) → NADP⁺. | Proton gradient across thylakoid membrane | ATP produced by ATP synthase (chemiosmosis). |
| 4. **NADPH Production** | Electrons reduce NADP⁺ to NADPH. | PSI, ferredoxin | NADPH carries reducing power for carbon fixation. |
| 5. **Carbon Fixation (Calvin Cycle)** | CO₂ is fixed into 3‑phosphoglycerate (3‑PGA) via Rubisco. | Ribulose‑1‑5‑phosphate (RuBP), ATP, NADPH | 3‑PGA is converted to glyceraldehyde‑3‑phosphate (G3P). |
| 6. **Glucose Synthesis** | G3P is rearranged, phosphorylated, and eventually forms glucose (C₆H₁₂O₆). | Enzymes of the Calvin cycle | Glucose (and other sugars) stored or used for growth. |
| 7. **Regeneration of RuBP** | Some G3P is used to regenerate RuBP, allowing the cycle to continue. | ATP, NADPH | Cycle repeats. |
### Key Points
- **Location**: All reactions occur in chloroplasts, specifically in the thylakoid membranes (light‑dependent reactions) and stroma (Calvin cycle).
- **Energy Flow**: Light energy → ATP + NADPH → CO₂ fixation → sugars.
- **Oxygen Production**: O₂ is a by‑product of water splitting; it
Question: What causes the greenhouse effect and how does it work?
Response:
<|channel|>analysis<|message|>Need explanation.<|end|><|start|>assistant<|channel|>final<|message|>### What Causes the Greenhouse Effect?
The greenhouse effect is a natural process that keeps Earth’s surface warm enough to support life. It is caused by certain gases in the atmosphere—called **greenhouse gases (GHGs)**—that absorb, trap, and re‑emit infrared (IR) radiation emitted by the planet. The main contributors are:
| Greenhouse Gas | Natural Sources | Anthropogenic (human) Sources | Global Concentration (ppm) |
|-----------------|-------------------|------------------------------|---------------------------|
| **Carbon dioxide (CO₂)** | Volcanoes, respiration, decomposition | Fossil‑fuel combustion, deforestation, cement production | ~420 ppm (2024) |
| **Methane (CH₄)** | Wetlands, termites, natural seepage | Livestock, rice paddies, landfills, oil & gas extraction | ~1.9 ppm |
| **Nitrous oxide (N₂O)** | Soil microbes, decomposition | Agricultural fertilizers, combustion | ~330 ppb |
| **Water vapor (H₂O)** | Evaporation from oceans, lakes, soil | Indirectly increased by warming (feedback) | 0.3–4 % (variable) |
| **Ozone (O₃)** | Photochemical reactions | Industrial emissions, vehicle exhaust | ~300 ppb |
| **Fluorinated gases (e.g., CFCs, HFCs, SF₆)** | None (synthetic) | Refrigerants, aerosols, industrial processes | <1 ppb (but very potent) |
> **Key point:** Water vapor is the most abundant greenhouse gas, but its concentration is largely controlled by temperature (a *feedback*). The other gases are *forcing agents* that directly change the radiative balance.
---
### How the Greenhouse Effect Works (Radiative Transfer)
1. **Solar Energy In**
- The Sun emits mostly visible and near‑infrared (NIR) radiation.
- About 30 % of this solar radiation is reflected back to space by clouds, ice, and the atmosphere.
- Roughly 70 % reaches Earth’s surface and is absorbed, warming the land, oceans, and atmosphere.
2. **Earth’s Thermal Emission**
- A warm Earth emits long‑wave (infrared) radiation, peaking around 10 µm.
- In
Question: Describe the structure and function of DNA.
Response:
<|channel|>analysis<|message|>Need concise description.<|end|><|start|>assistant<|channel|>final<|message|>**DNA (Deoxyribonucleic Acid)** is the hereditary material that carries the genetic instructions for building and maintaining an organism. Its structure and function can be understood in two complementary ways: the *molecular architecture* that makes it a stable, information‑carrying polymer, and the *biological roles* it plays in life.
---
## 1. Molecular Structure
| Feature | Description | Biological Significance |
|---------|-------------|--------------------------|
| **Back‑bone** | Alternating sugar (deoxyribose) and phosphate groups. | Provides chemical stability and a uniform “track” for base pairing. |
| **Bases** | Four nitrogenous bases: Adenine (A), Thymine (T), Cytosine (C), Guanine (G). | Encode genetic information; base pairs are complementary (A‑T, C‑G). |
| **Double helix** | Two complementary strands wound around each other. | Protects genetic material, allows replication and repair. |
| **Major & minor grooves** | Spaces between the strands. | Sites for protein binding (e.g., transcription factors, polymerases). |
| **Antiparallel orientation** | One strand runs 5’→3’, the other 3’→5’. | Essential for DNA polymerase activity during replication. |
| **Base‑pairing rules** | A pairs with T, C pairs with G. | Enables faithful copying of genetic information. |
| **Chromatin packaging** | DNA wrapped around histones → nucleosomes → higher‑order structures. | Condenses DNA into chromosomes, regulates gene expression. |
---
## 2. Biological Functions
| Function | How DNA accomplishes it | Key Players |
|----------|------------------------|-------------|
| **Genetic information storage** | Sequence of bases encodes proteins, RNAs, regulatory elements. | Genes, regulatory sequences |
| **Replication** | Semi‑conservative copying: each new duplex contains one old strand. | DNA polymerases, helicases, ligases |
| **Transcription** | DNA → messenger RNA (mRNA). | RNA polymerase, transcription factors |
| **Translation (indirect)** | mRNA → proteins. | Ribosomes, tRNA, aminoacyl‑tRNA synthetases |
| **Repair & maintenance** | Detect and fix mismatches, breaks, damage. | Exonucleases, endonucleases
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}