metadata

datasets:
  - togethercomputer/RedPajama-Data-V2
language:
  - de
library_name: transformers
license: other
tags:
  - fill-mask
  - masked-lm
  - long-context
  - modernbert
pipeline_tag: feature-extraction

ModernGBERT 134M

This is a German ModernBERT 134M language model trained from scratch using the ModernBERT codebase and the same German portion of RedPajama V2 as our LLäMmlein family. Find more details in our preprint!

Usage

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("LSX-UniWue/ModernGBERT_134M")

tokenizer = AutoTokenizer.from_pretrained("LSX-UniWue/ModernGBERT_134M")

Performance

We evaluated our model on the SuperGLEBer benchmark.