title: ModernGBERT: German-only 1B Encoder Model Trained from Scratch
summary: Despite the prominence of decoder-only language models, encoders remain
  crucial for resource-constrained applications. We introduce ModernGBERT (134M,
  1B), a fully transparent family of German encoder models trained from scratch,
  incorporating architectural innovations from ModernBERT. To evaluate the
  practical trade-offs of training encoders from scratch, we also present
  LL\"aMmlein2Vec (120M, 1B, 7B), a family of encoders derived from German
  decoder-only models via LLM2Vec. We benchmark all models on natural language
  understanding, text embedding, and long-context reasoning tasks, enabling a
  controlled comparison between dedicated encoders and converted decoders. Our
  results show that ModernGBERT 1B outperforms prior state-of-the-art German
  encoders as well as encoders adapted via LLM2Vec, with regard to performance
  and parameter-efficiency. All models, training data, checkpoints and code are
  publicly available, advancing the German NLP ecosystem with transparent,
  high-performance encoder models.
paper_authors._id: 683597e271c57c48d73d78a0
paper_authors.name: Anton Ehrmanntraut
paper_authors._id: 683597e271c57c48d73d78a1
paper_authors.name: Julia Wunderle
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T12:30:53.989Z
paper_authors._id: 61a36cba39c7c71f0bda980e
paper_authors.avatarUrl: /avatars/3a351b7ea31e82f350193415cc5b256e.svg
paper_authors.fullname: Julia Wunderle 
paper_authors.type: user
paper_authors.user: Julia287
paper_authors._id: 683597e271c57c48d73d78a2
paper_authors.name: Jan Pfister
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T12:13:08.065Z
paper_authors._id: 6070431e1a4c4d313032558b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bPLvEgvzsZJXWamPaU6ou.png
paper_authors.fullname: Jan Pfister
paper_authors.type: user
paper_authors.user: JanPf
paper_authors._id: 683597e271c57c48d73d78a3
paper_authors.name: Fotis Jannidis
paper_authors._id: 683597e271c57c48d73d78a4
paper_authors.name: Andreas Hotho
paper_title: ModernGBERT: German-only 1B Encoder Model Trained from Scratch
paper_summary: Despite the prominence of decoder-only language models, encoders remain
  crucial for resource-constrained applications. We introduce ModernGBERT (134M,
  1B), a fully transparent family of German encoder models trained from scratch,
  incorporating architectural innovations from ModernBERT. To evaluate the
  practical trade-offs of training encoders from scratch, we also present
  LL\"aMmlein2Vec (120M, 1B, 7B), a family of encoders derived from German
  decoder-only models via LLM2Vec. We benchmark all models on natural language
  understanding, text embedding, and long-context reasoning tasks, enabling a
  controlled comparison between dedicated encoders and converted decoders. Our
  results show that ModernGBERT 1B outperforms prior state-of-the-art German
  encoders as well as encoders adapted via LLM2Vec, with regard to performance
  and parameter-efficiency. All models, training data, checkpoints and code are
  publicly available, advancing the German NLP ecosystem with transparent,
  high-performance encoder models.
paper_ai_summary: ModernGBERT and LL\"aMmlein2Vec, new German encoder models, outperform existing models in terms of performance and parameter-efficiency across various NLP tasks.
paper_ai_keywords: encoder models ModernBERT LLM2Vec natural language understanding text embedding long-context reasoning parameter-efficiency German NLP