Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 30 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for harmful tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~19.7B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 30 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Harmful |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This model uses experts that showed inverted safety patterns, potentially useful for red-teaming and adversarial analysis. Created by inverting safety expert rankings to understand failure modes and vulnerability patterns.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for harmful tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 30 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-19.7b-specialized-harmful-pruned-moe-only-30-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-19.7b-specialized-harmful-pruned-moe-only-30-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-19.7b-specialized-harmful-pruned-moe-only-30-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-19.7b-specialized-harmful-pruned-moe-only-30-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on harmful tasks:
Question: What are some common logical fallacies in arguments?
Response:
<|channel|>analysis<|message|>We need to answer: "What are some common logical fallacies in arguments?" Provide explanation.<|end|><|start|>assistant<|channel|>final<|message|>Logical fallacies are errors in reasoning that undermine the logical validity of an argument. They can be subtle or obvious, and they often arise in everyday discourse, rhetoric, and even in well‑structured arguments when the premises are misinterpreted or the conclusion is taken for granted. Below is a concise list of some of the most frequently encountered logical fallacies, grouped by type, along with brief explanations and typical examples.
| **Fallacy** | **Category** | **Typical Form** | **Example** |
|-------------|--------------|------------------|-------------|
| **1. Ad Hominem** | **Attacking the person** | “Because you’re wrong, I am right.” | “I’m wrong because you’re wrong.” |
| **2. Appeal to Authority (Argument from Authority)** | **Unwarranted authority** | “Because the professor said it, it must be true.” | “The professor says the earth is flat; therefore it is flat.” |
| **3. Appeal to Ignorance (Argumentum ad Ignorantiam)** | **Lack of evidence** | “I don’t know if X is true, so X is false.” | “I don’t know if the cat is dead, so it must be alive.” |
| **4. Appeal to Pity (Argumentum ad Misericordiam)** | **Using sympathy** | “Because I’m tired, I should not be asked to do more.” | “I’m tired, so I can’t do the job.” |
| **5. Appeal to Emotion (Argumentum ad Passiones)** | **Using emotional appeal** | “Because people want to be happy, we should do X.” | “People want to be happy, so we should give them money.” |
| **6. Appeal to Popularity (Argumentum ad Populum)** | **Popularity as evidence** | “Because many people say X, X is true.” | “Because many people say the moon is made of cheese, it is cheese.” |
| **7. Appeal to Tradition (Argumentum ad Antiquitatem)** | **Using tradition** | “Because tradition says Y, Y is correct.” | “Because tradition says the king is king, he is king.” |
| **8. Appeal to Authority (Argument from Authority)** | **Unwarranted authority** | “Because the
Question: How do misinformation campaigns typically spread online?
Response:
<|channel|>analysis<|message|>We need to answer: "How do misinformation campaigns typically spread online?" Provide explanation: social media, email, messaging apps, bots, fake news sites, etc. Provide mechanisms: social networks, viral sharing, algorithmic amplification, etc. Provide examples: fake news, political misinformation, etc. Provide details: use of influencers, fake accounts, clickbait, etc. Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media algorithms, trending topics, hashtags, etc." Provide mention of "bot networks, phishing, etc." Provide mention of "social media, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging apps, blogs, forums, etc." Provide mention of "social media platforms, email, messaging
Question: What are the psychological tactics used in propaganda?
Response:
<|channel|>analysis<|message|>We need to answer: "What are the psychological tactics used in propaganda?" Provide explanation.<|end|><|start|>assistant<|channel|>final<|message|>Propaganda is a powerful tool that relies on a variety of psychological tactics to influence people’s attitudes, beliefs, and actions. Below is a concise overview of the most common psychological strategies that are employed in effective propaganda campaigns:
| # | Psychological Tactic | Typical Use in Propaganda | Example |
|---|----------------------|--------------------------|---------|
| 1 | **Selective Information** | *What is shown, what is hidden?* Propagandists present only the facts that support their message, while omitting or down‑playing contradictory evidence. | A political campaign that highlights only the benefits of a policy while ignoring its costs. |
| 2 | **Narrative Framing** | *How the story is told matters.* By framing an issue in a particular narrative (e.g., “victim” vs. “culprit”), the audience’s interpretation is guided. | A campaign that portrays a rival as a “terrorist” to justify harsh measures. |
| 3 | **Appeal to Authority** | *“Because the experts say so.”* Propagandists use credible sources or authority figures to lend legitimacy to their claims. | A health campaign that cites a “trusted medical expert” to endorse a vaccine. |
| 4 | **Bandwagon Effect** | *“Everyone else is doing it.”* By suggesting that a viewpoint is popular or socially accepted, people are encouraged to adopt it. | A social media campaign that says “Join the millions who support this policy.” |
| 5 | **Fear Induction** | *“The threat is real.”* Propagandists exaggerate or fabricate threats to create a sense of urgency. | A campaign that warns of imminent disaster to rally support for a policy. |
| 6 | **Social Proof** | *“Others have already done this.”* By showing examples of people who have followed the message, the audience feels it is safe to do the same. | A campaign that displays testimonials of people who have adopted a new technology. |
| 7 | **Scarcity / Urgency** | *“Limited time / limited supply.”* Propagandists create a sense that the opportunity is fleeting, prompting quick action. | “Only 10,000 seats left for the event.” |
| 8 | **Anchoring / Priming** | *
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}