Progress Report: Towards European LLMs
<|reference_start|>Progress Report: Towards European LLMs: We present preliminary results of the project OpenGPT-X. At present, the project has developed two multilingual LLMs designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, data processing techniques, tokenizer optimization, and training methodologies. The models demonstrate competitive performance across multilingual benchmarks, as evidenced by its performance on European versions of ARC, HellaSwag, MMLU, and TruthfulQA.<|reference_end|>
@article{ali2024teuken-7b-base,
    title={Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs},
    author={Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno
  Weber, Richard Rutmann, Charvi Jain, Max L"ubbering, Daniel Steinigen,
  Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat,
  Hammam Abdelwahab, Benny J"org Stein, Karl-Heinz Sylla, Pavel Denisov,
  Nicolo' Brandizzi, Qasid Saleem, Anirban Bhowmick, Lennard Helmer, Chelsea
  John, Pedro Ortiz Suarez, Malte Ostendorff, Alex Jude, Lalith Manjunath,
  Samuel Weinbach, Carolin Penke, Oleg Filatov, Shima Asaadi, Fabio Barth,
  Rafet Sifa, Fabian K"uch, Andreas Herten, Ren'e J"akel, Georg Rehm, Stefan
  Kesselheim, Joachim K"ohler, Nicolas Flores-Herr},
    journal={arXiv preprint arXiv:2410.03730},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03730},
    primaryClass={cs.CL cs.AI cs.LG}
}