title: Poro 34B and the Blessing of Multilinguality
summary: The pretraining of state-of-the-art large language models now requires
  trillions of words of text, which is orders of magnitude more than available
  for the vast majority of languages. While including text in more than one
  language is an obvious way to acquire more pretraining data, multilinguality is
  often seen as a curse, and most model training efforts continue to focus
  near-exclusively on individual large languages. We believe that multilinguality
  can be a blessing and that it should be possible to substantially improve over
  the capabilities of monolingual models for small languages through multilingual
  training. In this study, we introduce Poro 34B, a 34 billion parameter model
  trained for 1 trillion tokens of Finnish, English, and programming languages,
  and demonstrate that a multilingual training approach can produce a model that
  not only substantially advances over the capabilities of existing models for
  Finnish, but also excels in translation and is competitive in its class in
  generating English and programming languages. We release the model parameters,
  scripts, and data under open licenses at
  https://huggingface.co/LumiOpen/Poro-34B.
paper_authors._id: 660ccb8127b5ec2726612850
paper_authors.name: Risto Luukkonen
paper_authors._id: 660ccb8127b5ec2726612851
paper_authors.name: Jonathan Burdge
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-03T06:13:14.304Z
paper_authors._id: 6530b367471261a41245cfbb
paper_authors.avatarUrl: /avatars/7c7df91461068adeda05d59c7244d9d3.svg
paper_authors.fullname: jonabur
paper_authors.type: user
paper_authors.user: jonabur
paper_authors._id: 660ccb8127b5ec2726612852
paper_authors.name: Elaine Zosa
paper_authors._id: 660ccb8127b5ec2726612853
paper_authors.name: Aarne Talman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:29:03.066Z
paper_authors._id: 6437ac261ee0e43f14d3b27f
paper_authors.avatarUrl: /avatars/5b1d7db9a4f16ba8cf71f982daee4c81.svg
paper_authors.fullname: Aarne Talman
paper_authors.type: user
paper_authors.user: talman-fi
paper_authors._id: 660ccb8127b5ec2726612854
paper_authors.name: Ville Komulainen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T05:51:53.803Z
paper_authors._id: 63f4d05f121f98947078baa6
paper_authors.avatarUrl: /avatars/02a16a535f7aa90b6b312388293b2bbd.svg
paper_authors.fullname: Ville Komulainen
paper_authors.type: user
paper_authors.user: Villekom
paper_authors._id: 660ccb8127b5ec2726612855
paper_authors.name: Väinö Hatanpää
paper_authors._id: 660ccb8127b5ec2726612856
paper_authors.name: Peter Sarlin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-04T17:35:45.232Z
paper_authors._id: 660ee4e6caa442b9045428f9
paper_authors.avatarUrl: /avatars/765d0b583749b48456f2ba1d8ea47dab.svg
paper_authors.fullname: Peter Sarlin
paper_authors.type: user
paper_authors.user: psarlin
paper_authors._id: 660ccb8127b5ec2726612857
paper_authors.name: Sampo Pyysalo
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-03T07:16:44.855Z
paper_authors._id: 602fb2d93d343696d96ef287
paper_authors.avatarUrl: /avatars/a78ce59506a416e3df1d283a13e5aefa.svg
paper_authors.fullname: Sampo Pyysalo
paper_authors.type: user
paper_authors.user: spyysalo
paper_title: Poro 34B and the Blessing of Multilinguality
paper_summary: The pretraining of state-of-the-art large language models now requires
  trillions of words of text, which is orders of magnitude more than available
  for the vast majority of languages. While including text in more than one
  language is an obvious way to acquire more pretraining data, multilinguality is
  often seen as a curse, and most model training efforts continue to focus
  near-exclusively on individual large languages. We believe that multilinguality
  can be a blessing and that it should be possible to substantially improve over
  the capabilities of monolingual models for small languages through multilingual
  training. In this study, we introduce Poro 34B, a 34 billion parameter model
  trained for 1 trillion tokens of Finnish, English, and programming languages,
  and demonstrate that a multilingual training approach can produce a model that
  not only substantially advances over the capabilities of existing models for
  Finnish, but also excels in translation and is competitive in its class in
  generating English and programming languages. We release the model parameters,
  scripts, and data under open licenses at
  https://huggingface.co/LumiOpen/Poro-34B.
paper_ai_keywords: pretraining large language models trillions of words of text multilinguality monolingual models Poro 34B 34 billion parameter model 1 trillion tokens Finnish English programming languages translation