pdjohn/EuroBERT-610m-term-sense

T-EBERT: German IR Term Sense Disambiguation

Fine-tuned EuroBERT-610M for disambiguating International Relations term senses in German.

Model Description

This model distinguishes between specialized (International Relations) and colloquial usage of German political science terms:

Input: "Die internationale Norm verbietet den Einsatz von Gewalt."
Output: IR sense (specialized usage)

Input: "Das entspricht nicht den technischen Normen in Deutschland."
Output: Colloquial sense (general usage)

Performance

• F1 Score: 0.922 on held-out test set
• Training / Test distribution: 70% IR sense, 30% colloquial (natural corpus distribution)

  Term	F1 Score
  Entspannung	0.967
  Intervention	0.949
  Integration	0.933
  Kooperation	0.926
  Norm	0.831
  Regime	0.828

Intended Use

• German IR corpus analysis and text mining
• Political science document classification
• Automated terminology extraction in specialized texts
• Production deployment on naturally distributed data

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "pdjohn/T-EBERT-term-sense-german"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Classify a sentence
sentence = "Die internationale Norm verbietet den Einsatz von Gewalt."
inputs = tokenizer(sentence, return_tensors="pt", truncation=True, padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=1).item()

# Interpretation
labels = {0: "Colloquial", 1: "IR Sense"}
print(f"Prediction: {labels[prediction]}")

Training Data

• Corpus: German political science texts from IR literature
• Size: 934 sentences (training) + 234 sentences (test)
• Distribution: 70% IR sense, 30% colloquial (natural)
• Terms: Norm, Kooperation, Regime, Integration, Intervention, Entspannung
• Annotation: Manual expert annotation

Training Procedure

• Base model: EuroBERT-610M
• Fine-tuning method: LoRA (Low-Rank Adaptation)
  • Rank (r): 8
  • Alpha: 16
  • Target modules: q_proj, k_proj, v_proj
  • Dropout: 0.05
• Epochs: 5
• Batch size: 8
• Learning rate: 1e-4
• Optimizer: AdamW with cosine learning rate schedule

Limitations

• Trained on imbalanced data (70/30 split)
• Performance may drop on datasets with different distributions
• see T-EBERT-Balanced for a variant with distribution robustness

Acknowledgements

This work was conducted as part of the Tracing International Institutions and Behavior (TIIB) project at TU Darmstadt.

Affiliation: Chair of Transnational Governance, Department of Political Science, Technische Universität Darmstadt

We thank the TIIB project team for their support and the manual annotation of the training corpus.