title: LLäMmlein: Compact and Competitive German-Only Language Models from
    Scratch
summary: We create two German-only decoder models, LL\"aMmlein 120M and 1B,
  transparently from scratch and publish them, along with the training data, for
  the German NLP research community to use. The model training involved several
  key steps, including extensive data preprocessing, the creation of a custom
  German tokenizer, the training itself, as well as the evaluation of the final
  models on various benchmarks. Throughout the training process, multiple
  checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor
  the models' learning dynamics. Compared to state-of-the-art models on the
  SuperGLEBer benchmark, both LL\"aMmlein models performed competitively,
  consistently matching or surpassing models with similar parameter sizes. The
  results show that the models' quality scales with size as expected, but
  performance improvements on some tasks plateaued early, offering valuable
  insights into resource allocation for future model development.
paper_authors._id: 673c3f6c19cbbe3091dc6ad7
paper_authors.name: Jan Pfister
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T16:38:08.396Z
paper_authors._id: 6070431e1a4c4d313032558b
paper_authors.avatarUrl: /avatars/142102e7e14886a0ff34af7959ca2f8c.svg
paper_authors.fullname: Jan Pfister
paper_authors.type: user
paper_authors.user: JanPf
paper_authors._id: 673c3f6c19cbbe3091dc6ad8
paper_authors.name: Julia Wunderle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T16:37:46.563Z
paper_authors._id: 61a36cba39c7c71f0bda980e
paper_authors.avatarUrl: /avatars/3a351b7ea31e82f350193415cc5b256e.svg
paper_authors.fullname: Julia Wunderle 
paper_authors.type: user
paper_authors.user: Julia287
paper_authors._id: 673c3f6c19cbbe3091dc6ad9
paper_authors.name: Andreas Hotho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T16:37:40.241Z
paper_authors._id: 670d2ac02a2ae377cf4aa56e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CDM7_7XmFF-1pCqsUkrBB.png
paper_authors.fullname: Andreas Hotho
paper_authors.type: user
paper_authors.user: arthur3131
paper_title: LLäMmlein: Compact and Competitive German-Only Language Models from
    Scratch
paper_summary: We create two German-only decoder models, LL\"aMmlein 120M and 1B,
  transparently from scratch and publish them, along with the training data, for
  the German NLP research community to use. The model training involved several
  key steps, including extensive data preprocessing, the creation of a custom
  German tokenizer, the training itself, as well as the evaluation of the final
  models on various benchmarks. Throughout the training process, multiple
  checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor
  the models' learning dynamics. Compared to state-of-the-art models on the
  SuperGLEBer benchmark, both LL\"aMmlein models performed competitively,
  consistently matching or surpassing models with similar parameter sizes. The
  results show that the models' quality scales with size as expected, but
  performance improvements on some tasks plateaued early, offering valuable
  insights into resource allocation for future model development.
paper_ai_keywords: decoder models inverted index data preprocessing custom tokenizer SuperGLEBer benchmark learning dynamics parameter sizes quality scaling performance improvements resource allocation