Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 32 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for harmful tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~20.9B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 32 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Harmful |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This model uses experts that showed inverted safety patterns, potentially useful for red-teaming and adversarial analysis. Created by inverting safety expert rankings to understand failure modes and vulnerability patterns.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for harmful tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 32 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.9b-specialized-harmful-pruned-moe-only-32-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on harmful tasks:
Question: What are some common logical fallacies in arguments?
Response:
<|channel|>analysis<|message|>Need to list common logical fallacies. Provide explanation.<|end|><|start|>assistant<|channel|>final<|message|>### Common Logical Fallacies in Arguments
| # | Fallacy | What It Looks Like | Why It’s Problematic |
|---|---------|--------------------|----------------------|
| 1 | **Ad Hominem** | Attacking the person instead of the argument (e.g., “You’re wrong because you’re a liar.”) | Dismisses the argument’s merits and shifts focus to irrelevant personal traits. |
| 2 | **Straw Man** | Misrepresenting or oversimplifying someone’s position to make it easier to attack (e.g., “He says we should reduce taxes, so he wants to give everyone free money.”) | Creates a false target, so the rebuttal doesn’t address the real issue. |
| 3 | **Appeal to Authority** | Claiming something is true because an authority says so, without examining evidence (e.g., “The CEO says this product is safe, so it must be.”) | Authority can be wrong, biased, or irrelevant; the claim still needs evidence. |
| 4 | **Appeal to Emotion (Pathos)** | Using fear, pity, or anger to win an argument (e.g., “If you don’t vote for this law, innocent children will suffer!”) | Emotions can override rational analysis and lead to decisions based on feelings rather than facts. |
| 5 | **False Dilemma / Black‑and‑White** | Presenting only two options when more exist (e.g., “Either we ban all cars or we accept endless pollution.”) | Limits the discussion and ignores nuanced or middle‑ground solutions. |
| 6 | **Circular Reasoning (Begging the Question)** | Using the conclusion as a premise (e.g., “The Bible is true because it says so in the Bible.”) | Provides no independent support; the argument loops back on itself. |
| 7 | **Hasty Generalization** | Drawing a broad conclusion from a small or unrepresentative sample (e.g., “I met two rude tourists; therefore all tourists are rude.”) | Overlooks variability and can lead to stereotypes. |
| 8 | **Post Hoc (False Cause)** | Assuming that because B follows A, A caused B (e.g., “The lights went out after the storm; the storm caused the outage.”) | Correlation does not equal causation; other
Question: How do misinformation campaigns typically spread online?
Response:
<|channel|>analysis<|message|>Need explain mechanisms: bots, echo chambers, algorithms, sensational content, amplification, social media, deepfakes, etc.<|end|><|start|>assistant<|channel|>final<|message|>### 1. The “Pipeline” of a Misinformation Campaign
| Stage | What Happens | Typical Tools / Actors |
|-------|--------------|------------------------|
| **Ideation & Production** | A narrative is crafted (political, health, financial, etc.). | Paid writers, political operatives, fringe groups, AI‑generated text. |
| **Seeding** | The story is first posted on a few accounts or forums. | Bots, “sockpuppet” accounts, paid influencers, niche subreddits. |
| **Amplification** | The content is shared, reshared, and promoted. | Social‑media algorithms, paid ads, trending‑topic hijacking, cross‑platform reposting. |
| **Echo‑Chamber Reinforcement** | Like‑minded users consume, comment, and share, creating a feedback loop. | Community groups, algorithmic filtering, “filter bubbles.” |
| **Legitimization** | Fact‑checkers, experts, or mainstream outlets are co-opted or misquoted. | Deepfakes, doctored images, “expert” videos, fake citations. |
| **Saturation & Persistence** | The narrative becomes entrenched, often resurfacing in new contexts. | Scheduled reposts, automated bots, “shadow‑banning” of counter‑content. |
---
### 2. Key Mechanisms that Make It Work
| Mechanism | How It Works | Why It’s Effective |
|-----------|--------------|--------------------|
| **Bots & Automated Accounts** | Mass‑post identical or slightly varied messages. | They can out‑number real users, creating the illusion of widespread support. |
| **Sockpuppets & Fake Personas** | Human operators create multiple accounts that appear authentic. | They can engage in debates, “discredit” opponents, and seed content organically. |
| **Algorithmic Amplification** | Platforms reward engagement (likes, shares, comments). | Misinformation that triggers strong emotions (fear, anger, curiosity) gets pushed to more feeds. |
| **Echo Chambers & Filter Bubbles** | Users follow like‑minded accounts and see only reinforcing content. | Reduces exposure to corrective information and strengthens belief. |
| **Deepfakes & Synthetic Media** | AI‑generated videos or audio that look real. |
Question: What are the psychological tactics used in propaganda?
Response:
<|channel|>analysis<|message|>Need to explain tactics.<|end|><|start|>assistant<|channel|>final<|message|>Propaganda is a systematic attempt to shape attitudes, beliefs, and behaviors by manipulating information and emotional responses. Psychologists and communication scholars have identified a number of recurring tactics that make propaganda especially effective. Below is a concise, organized overview of the most common psychological mechanisms, along with brief examples and the underlying theory that explains why they work.
| # | Tactic | Psychological Principle | How It Works | Typical Example |
|---|--------|------------------------|--------------|-----------------|
| 1 | **Repetition** | **Priming & Mere Exposure Effect** | Repeating a message over and over makes it more familiar, which increases liking and perceived truth. | “Make America Great Again” slogan repeated in ads, rallies, and media. |
| 2 | **Bandwagon** | **Social Proof** | People are more likely to adopt an idea if they believe many others do. | “Millions of Americans support…”. |
| 3 | **Fear‑Appeal** | **Arousal‑Induced Persuasion** | Fear increases arousal, which heightens attention and memory for the message. | “If we don’t act, our children will die.” |
| 4 | **Scapegoating** | **In‑group/Out‑group Bias** | Blaming a specific group for problems reinforces in‑group cohesion and justifies hostility. | “The immigrants are stealing jobs.” |
| 5 | **Simplification / Black‑and‑White Framing** | **Cognitive Load Reduction** | Complex issues are reduced to a simple dichotomy, making them easier to process and remember. | “It’s either us or them.” |
| 6 | **Emotional Storytelling** | **Narrative Transportation** | Stories engage empathy and transport the audience, reducing counter‑arguing. | Personal testimonies of “victims” of a policy. |
| 7 | **Authority Endorsement** | **Authority Bias** | Endorsements from perceived experts or leaders increase credibility. | “The President says…”. |
| 8 | **Selective Exposure / Confirmation Bias** | **Motivated Reasoning** | Presenting only evidence that supports the desired conclusion while ignoring contradictory data. | “All the evidence shows…”. |
| 9 | **Loaded Language / Framing** | **Framing Effect** | The way information is worded (e.g., “freedom fighters” vs
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}