title: FinGPT: Large Generative Models for a Small Language
summary: Large language models (LLMs) excel in many tasks in NLP and beyond, but most
  open models have very limited coverage of smaller languages and LLM work tends
  to focus on languages where nearly unlimited data is available for pretraining.
  In this work, we study the challenges of creating LLMs for Finnish, a language
  spoken by less than 0.1% of the world population. We compile an extensive
  dataset of Finnish combining web crawls, news, social media and eBooks. We
  pursue two approaches to pretrain models: 1) we train seven monolingual models
  from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the
  pretraining of the multilingual BLOOM model on a mix of its original training
  data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
  For model evaluation, we introduce FIN-bench, a version of BIG-bench with
  Finnish tasks. We also assess other model qualities such as toxicity and bias.
  Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.
paper_authors._id: 65519918a43b13ee3479719e
paper_authors.name: Risto Luukkonen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:51:53.285Z
paper_authors._id: 60fab40cd0b1858cd139b745
paper_authors.avatarUrl: /avatars/678b84ae7dff1115cff40efbe366ca7d.svg
paper_authors.fullname: Risto Luukkonen
paper_authors.type: user
paper_authors.user: rluukkon
paper_authors._id: 65519918a43b13ee3479719f
paper_authors.name: Ville Komulainen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:01.554Z
paper_authors._id: 63f4d05f121f98947078baa6
paper_authors.avatarUrl: /avatars/02a16a535f7aa90b6b312388293b2bbd.svg
paper_authors.fullname: Ville Komulainen
paper_authors.type: user
paper_authors.user: Villekom
paper_authors._id: 65519918a43b13ee347971a0
paper_authors.name: Jouni Luoma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:08.385Z
paper_authors._id: 63067e7414e7d8892aadb008
paper_authors.avatarUrl: /avatars/d5c2ab86bd5df07dabc055b77cf09927.svg
paper_authors.fullname: Jouni Luoma
paper_authors.type: user
paper_authors.user: Joffe
paper_authors._id: 65519918a43b13ee347971a1
paper_authors.name: Anni Eskelinen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:15.469Z
paper_authors._id: 63ecae3454bba6781516a98b
paper_authors.avatarUrl: /avatars/9d270daa837a8a9b01b0d312eb0e184d.svg
paper_authors.fullname: Anni Eskelinen
paper_authors.type: user
paper_authors.user: annieske
paper_authors._id: 65519918a43b13ee347971a2
paper_authors.name: Jenna Kanerva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:21.955Z
paper_authors._id: 618e380849baf26530e19574
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636710617882-618e380849baf26530e19574.png
paper_authors.fullname: Jenna Kanerva
paper_authors.type: user
paper_authors.user: jmnybl
paper_authors._id: 65519918a43b13ee347971a3
paper_authors.name: Hanna-Mari Kupari
paper_authors._id: 65519918a43b13ee347971a4
paper_authors.name: Filip Ginter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:39.230Z
paper_authors._id: 618a6b0bd128cf918549eff8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676473724392-618a6b0bd128cf918549eff8.jpeg
paper_authors.fullname: Filip Ginter
paper_authors.type: user
paper_authors.user: fginter
paper_authors._id: 65519918a43b13ee347971a5
paper_authors.name: Veronika Laippala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:46.290Z
paper_authors._id: 6180f823067f702bd7415c03
paper_authors.avatarUrl: /avatars/726112779125678e0ca8e77423bc9eb4.svg
paper_authors.fullname: Veronika Laippala
paper_authors.type: user
paper_authors.user: VeronikaLaippala
paper_authors._id: 65519918a43b13ee347971a6
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:52:53.430Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65519918a43b13ee347971a7
paper_authors.name: Aleksandra Piktus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:53:00.628Z
paper_authors._id: 5f5f4f093c67af20d9945a6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1600081648206-noauth.jpeg
paper_authors.fullname: Aleksandra Piktus
paper_authors.type: user
paper_authors.user: ola13
paper_authors._id: 65519918a43b13ee347971a8
paper_authors.name: Thomas Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T16:56:20.423Z
paper_authors._id: 61140e35d08630d2676c9829
paper_authors.avatarUrl: /avatars/3d6e4b4d02eda7b5ef28e1cc0fb8e08a.svg
paper_authors.fullname: Thomas Wang
paper_authors.type: user
paper_authors.user: TimeRobber
paper_authors._id: 65519918a43b13ee347971a9
paper_authors.name: Nouamane Tazi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:53:34.607Z
paper_authors._id: 5ff8c9f4b2035d9a81a859f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652134289581-5ff8c9f4b2035d9a81a859f7.jpeg
paper_authors.fullname: Nouamane Tazi
paper_authors.type: user
paper_authors.user: nouamanetazi
paper_authors._id: 65519918a43b13ee347971aa
paper_authors.name: Teven Le Scao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:53:45.537Z
paper_authors._id: 5e67bed6100906368940747b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583858339980-5e67bed6100906368940747b.jpeg
paper_authors.fullname: Teven Le Scao
paper_authors.type: user
paper_authors.user: teven
paper_authors._id: 65519918a43b13ee347971ab
paper_authors.name: Thomas Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:54:13.866Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_authors._id: 65519918a43b13ee347971ac
paper_authors.name: Osma Suominen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:55:06.920Z
paper_authors._id: 633dc4e6c6e26ad2575e1799
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633dc4e6c6e26ad2575e1799/LuRAZ_B_tLYJl0uzy0OkY.jpeg
paper_authors.fullname: Osma Suominen
paper_authors.type: user
paper_authors.user: osma
paper_authors._id: 65519918a43b13ee347971ad
paper_authors.name: Samuli Sairanen
paper_authors._id: 65519918a43b13ee347971ae
paper_authors.name: Mikko Merioksa
paper_authors._id: 65519918a43b13ee347971af
paper_authors.name: Jyrki Heinonen
paper_authors._id: 65519918a43b13ee347971b0
paper_authors.name: Aija Vahtola
paper_authors._id: 65519918a43b13ee347971b1
paper_authors.name: Samuel Antao
paper_authors._id: 65519918a43b13ee347971b2
paper_authors.name: Sampo Pyysalo
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-02T08:08:25.122Z
paper_authors._id: 602fb2d93d343696d96ef287
paper_authors.avatarUrl: /avatars/a78ce59506a416e3df1d283a13e5aefa.svg
paper_authors.fullname: Sampo Pyysalo
paper_authors.type: user
paper_authors.user: spyysalo
paper_title: FinGPT: Large Generative Models for a Small Language
paper_summary: Large language models (LLMs) excel in many tasks in NLP and beyond, but most
  open models have very limited coverage of smaller languages and LLM work tends
  to focus on languages where nearly unlimited data is available for pretraining.
  In this work, we study the challenges of creating LLMs for Finnish, a language
  spoken by less than 0.1% of the world population. We compile an extensive
  dataset of Finnish combining web crawls, news, social media and eBooks. We
  pursue two approaches to pretrain models: 1) we train seven monolingual models
  from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the
  pretraining of the multilingual BLOOM model on a mix of its original training
  data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
  For model evaluation, we introduce FIN-bench, a version of BIG-bench with
  Finnish tasks. We also assess other model qualities such as toxicity and bias.
  Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.
paper_ai_keywords: Large language models (LLMs) natural language processing (NLP) multilingual BLOOM model monolingual models yielding pretrained models pretraining model evaluation BIG-bench toxicity bias FIN-bench BLUUMI FinGPT