Poro 34B and the Blessing of Multilinguality
The pretraining of state-of-the-art large language models now requires
trillions of words of text, which is orders of magnitude more than available
for the vast majority of languages. While including text in more than one
language is an obvious way to acquire more pretraining data, multilinguality is
often seen as a curse, and most model training efforts continue to focus
near-exclusively on individual large languages. We believe that multilinguality
can be a blessing and that it should be possible to substantially improve over
the capabilities of monolingual models for small languages through multilingual
training. In this study, we introduce Poro 34B, a 34 billion parameter model
trained for 1 trillion tokens of Finnish, English, and programming languages,
and demonstrate that a multilingual training approach can produce a model that
not only substantially advances over the capabilities of existing models for
Finnish, but also excels in translation and is competitive in its class in
generating English and programming languages. We release the model parameters,
scripts, and data under open licenses at
https://huggingface.co/LumiOpen/Poro-34B.
660ccb8127b5ec2726612850
Risto Luukkonen
660ccb8127b5ec2726612851
Jonathan Burdge
extracted_confirmed
2024-04-03T06:13:14.304Z
6530b367471261a41245cfbb
/avatars/7c7df91461068adeda05d59c7244d9d3.svg
jonabur
user
jonabur
660ccb8127b5ec2726612852
Elaine Zosa
660ccb8127b5ec2726612853
Aarne Talman
claimed_verified
2024-04-03T09:29:03.066Z
6437ac261ee0e43f14d3b27f
/avatars/5b1d7db9a4f16ba8cf71f982daee4c81.svg
Aarne Talman
user
talman-fi
660ccb8127b5ec2726612854
Ville Komulainen
claimed_verified
2024-04-11T05:51:53.803Z
63f4d05f121f98947078baa6
/avatars/02a16a535f7aa90b6b312388293b2bbd.svg
Ville Komulainen
user
Villekom
660ccb8127b5ec2726612855
Väinö Hatanpää
660ccb8127b5ec2726612856
Peter Sarlin
extracted_pending
2024-04-04T17:35:45.232Z
660ee4e6caa442b9045428f9
/avatars/765d0b583749b48456f2ba1d8ea47dab.svg
Peter Sarlin
user
psarlin
660ccb8127b5ec2726612857
Sampo Pyysalo
extracted_confirmed
2024-04-03T07:16:44.855Z
602fb2d93d343696d96ef287
/avatars/a78ce59506a416e3df1d283a13e5aefa.svg
Sampo Pyysalo
user
spyysalo
Poro 34B and the Blessing of Multilinguality
The pretraining of state-of-the-art large language models now requires
trillions of words of text, which is orders of magnitude more than available
for the vast majority of languages. While including text in more than one
language is an obvious way to acquire more pretraining data, multilinguality is
often seen as a curse, and most model training efforts continue to focus
near-exclusively on individual large languages. We believe that multilinguality
can be a blessing and that it should be possible to substantially improve over
the capabilities of monolingual models for small languages through multilingual
training. In this study, we introduce Poro 34B, a 34 billion parameter model
trained for 1 trillion tokens of Finnish, English, and programming languages,
and demonstrate that a multilingual training approach can produce a model that
not only substantially advances over the capabilities of existing models for
Finnish, but also excels in translation and is competitive in its class in
generating English and programming languages. We release the model parameters,
scripts, and data under open licenses at
https://huggingface.co/LumiOpen/Poro-34B.
A multilingual training approach on a large language model improves capabilities for small languages, translation, and generation in multiple languages.
large language models pretraining multilinguality monolingual models parameter model training transliteration translation generation