LLäMmlein: Compact and Competitive German-Only Language Models from
  Scratch
We create two German-only decoder models, LL\"aMmlein 120M and 1B,
transparently from scratch and publish them, along with the training data, for
the German NLP research community to use. The model training involved several
key steps, including extensive data preprocessing, the creation of a custom
German tokenizer, the training itself, as well as the evaluation of the final
models on various benchmarks. Throughout the training process, multiple
checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor
the models' learning dynamics. Compared to state-of-the-art models on the
SuperGLEBer benchmark, both LL\"aMmlein models performed competitively,
consistently matching or surpassing models with similar parameter sizes. The
results show that the models' quality scales with size as expected, but
performance improvements on some tasks plateaued early, offering valuable
insights into resource allocation for future model development.
673c3f6c19cbbe3091dc6ad7
Jan Pfister
admin_assigned
2024-11-19T16:38:08.396Z
6070431e1a4c4d313032558b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bPLvEgvzsZJXWamPaU6ou.png
Jan Pfister
user
JanPf
673c3f6c19cbbe3091dc6ad8
Julia Wunderle
admin_assigned
2024-11-19T16:37:46.563Z
61a36cba39c7c71f0bda980e
/avatars/3a351b7ea31e82f350193415cc5b256e.svg
Julia Wunderle 
user
Julia287
673c3f6c19cbbe3091dc6ad9
Andreas Hotho
admin_assigned
2024-11-19T16:37:40.241Z
670d2ac02a2ae377cf4aa56e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CDM7_7XmFF-1pCqsUkrBB.png
Andreas Hotho
user
arthur3131
LLäMmlein: Compact and Competitive German-Only Language Models from
  Scratch
We create two German-only decoder models, LL\"aMmlein 120M and 1B,
transparently from scratch and publish them, along with the training data, for
the German NLP research community to use. The model training involved several
key steps, including extensive data preprocessing, the creation of a custom
German tokenizer, the training itself, as well as the evaluation of the final
models on various benchmarks. Throughout the training process, multiple
checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor
the models' learning dynamics. Compared to state-of-the-art models on the
SuperGLEBer benchmark, both LL\"aMmlein models performed competitively,
consistently matching or surpassing models with similar parameter sizes. The
results show that the models' quality scales with size as expected, but
performance improvements on some tasks plateaued early, offering valuable
insights into resource allocation for future model development.
Two German-only decoder models, LL\"aMmlein 120M and 1B, were developed from scratch and achieved competitive performance on the SuperGLEBer benchmark, providing insights into the scalability and resource allocation for such models.
decoder models SuperGLEBer benchmark tokenizer data preprocessing model training evaluation learning dynamics