title: CroissantLLM: A Truly Bilingual French-English Language Model
summary: We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T
  English and French tokens, to bring to the research and industrial community a
  high-performance, fully open-sourced bilingual model that runs swiftly on
  consumer-grade local hardware. To that end, we pioneer the approach of training
  an intrinsically bilingual model with a 1:1 English-to-French pretraining data
  ratio, a custom tokenizer, and bilingual finetuning datasets. We release the
  training dataset, notably containing a French split with manually curated,
  high-quality, and varied data sources. To assess performance outside of
  English, we craft a novel benchmark, FrenchBench, consisting of an array of
  classification and generation tasks, covering various orthogonal aspects of
  model performance in the French Language. Additionally, rooted in transparency
  and to foster further Large Language Model research, we release codebases, and
  dozens of checkpoints across various model sizes, training data distributions,
  and training steps, as well as fine-tuned Chat models, and strong translation
  models. We evaluate our model through the FMTI framework, and validate 81 % of
  the transparency criteria, far beyond the scores of even most open initiatives.
  This work enriches the NLP landscape, breaking away from previous
  English-centric work in order to strengthen our understanding of
  multilinguality in language models.
paper_authors._id: 65bc67dce168a9cc4bc436b3
paper_authors.name: Manuel Faysse
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-27T10:03:04.135Z
paper_authors._id: 60f2e021adf471cbdf8bb660
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
paper_authors.fullname: Manuel Faysse
paper_authors.type: user
paper_authors.user: manu
paper_authors._id: 65bc67dce168a9cc4bc436b4
paper_authors.name: Patrick Fernandes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:31:07.251Z
paper_authors._id: 6354206b12edd0ed5dc235f4
paper_authors.avatarUrl: /avatars/c2f5872eb4cad979635293c0af47db03.svg
paper_authors.fullname: Patrick Fernandes
paper_authors.type: user
paper_authors.user: patricksf
paper_authors._id: 65bc67dce168a9cc4bc436b5
paper_authors.name: Nuno Guerreiro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:31:25.381Z
paper_authors._id: 61f06c8ba57920a251ec1a5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61f06c8ba57920a251ec1a5c/TmmSN9xWm1noLeAtsUSRk.jpeg
paper_authors.fullname: Nuno Guerreiro
paper_authors.type: user
paper_authors.user: nunonmg
paper_authors._id: 65bc67dce168a9cc4bc436b6
paper_authors.name: António Loison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:31:33.241Z
paper_authors._id: 61b3caeee49318df54457dc8
paper_authors.avatarUrl: /avatars/2f22de54b67c7ec76a5e3f07318b0916.svg
paper_authors.fullname: António Loison
paper_authors.type: user
paper_authors.user: antoloison
paper_authors._id: 65bc67dce168a9cc4bc436b7
paper_authors.name: Duarte Alves
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:31:40.248Z
paper_authors._id: 64132452d8a418df415a6ded
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64132452d8a418df415a6ded/qkjL5G89uldHUXlCI3n4f.jpeg
paper_authors.fullname: Duarte Alves
paper_authors.type: user
paper_authors.user: DuarteMRAlves
paper_authors._id: 65bc67dce168a9cc4bc436b8
paper_authors.name: Caio Corro
paper_authors._id: 65bc67dce168a9cc4bc436b9
paper_authors.name: Nicolas Boizard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:00:42.359Z
paper_authors._id: 62be186a5f59ff2320e6e32b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
paper_authors.fullname: Nicolas-BZRD
paper_authors.type: user
paper_authors.user: Nicolas-BZRD
paper_authors._id: 65bc67dce168a9cc4bc436ba
paper_authors.name: João Alves
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T11:18:37.546Z
paper_authors._id: 61ae32477b23f65fc71a62e4
paper_authors.avatarUrl: /avatars/3acf15ac95847c0d55205f16cba7b0f8.svg
paper_authors.fullname: João Alves
paper_authors.type: user
paper_authors.user: joaomiguel26
paper_authors._id: 65bc67dce168a9cc4bc436bb
paper_authors.name: Ricardo Rei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T11:10:31.846Z
paper_authors._id: 60d9a9791fa5d458da77754d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624887555363-60d9a9791fa5d458da77754d.jpeg
paper_authors.fullname: Ricardo Rei
paper_authors.type: user
paper_authors.user: RicardoRei
paper_authors._id: 65bc67dce168a9cc4bc436bc
paper_authors.name: Pedro Martins
paper_authors._id: 65bc67dce168a9cc4bc436bd
paper_authors.name: Antoni Bigata Casademunt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T11:25:23.341Z
paper_authors._id: 640777812e309e65452491dd
paper_authors.avatarUrl: /avatars/01b927675e78f212408168522f65fe36.svg
paper_authors.fullname: Antoni Bigata
paper_authors.type: user
paper_authors.user: toninio19
paper_authors._id: 65bc67dce168a9cc4bc436be
paper_authors.name: François Yvon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:32:17.706Z
paper_authors._id: 62ab10f04bd2ebf5dbad205c
paper_authors.avatarUrl: /avatars/65356b3b057159cc67a86efb26b53486.svg
paper_authors.fullname: François Yvon
paper_authors.type: user
paper_authors.user: fyvo
paper_authors._id: 65bc67dce168a9cc4bc436bf
paper_authors.name: André Martins
paper_authors._id: 65bc67dce168a9cc4bc436c0
paper_authors.name: Gautier Viaud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:33:15.397Z
paper_authors._id: 618a65e17304dc918c6602ff
paper_authors.avatarUrl: /avatars/8af1112094169da80c65e24ab71c7e59.svg
paper_authors.fullname: Gautier Viaud
paper_authors.type: user
paper_authors.user: gautierviaud
paper_authors._id: 65bc67dce168a9cc4bc436c1
paper_authors.name: Céline Hudelot
paper_authors._id: 65bc67dce168a9cc4bc436c2
paper_authors.name: Pierre Colombo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:33:37.095Z
paper_authors._id: 644a900e3a619fe72b14af0f
paper_authors.avatarUrl: /avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
paper_authors.fullname: Colombo
paper_authors.type: user
paper_authors.user: PierreColombo
paper_title: CroissantLLM: A Truly Bilingual French-English Language Model
paper_summary: We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T
  English and French tokens, to bring to the research and industrial community a
  high-performance, fully open-sourced bilingual model that runs swiftly on
  consumer-grade local hardware. To that end, we pioneer the approach of training
  an intrinsically bilingual model with a 1:1 English-to-French pretraining data
  ratio, a custom tokenizer, and bilingual finetuning datasets. We release the
  training dataset, notably containing a French split with manually curated,
  high-quality, and varied data sources. To assess performance outside of
  English, we craft a novel benchmark, FrenchBench, consisting of an array of
  classification and generation tasks, covering various orthogonal aspects of
  model performance in the French Language. Additionally, rooted in transparency
  and to foster further Large Language Model research, we release codebases, and
  dozens of checkpoints across various model sizes, training data distributions,
  and training steps, as well as fine-tuned Chat models, and strong translation
  models. We evaluate our model through the FMTI framework, and validate 81 % of
  the transparency criteria, far beyond the scores of even most open initiatives.
  This work enriches the NLP landscape, breaking away from previous
  English-centric work in order to strengthen our understanding of
  multilinguality in language models.
paper_ai_summary: CroissantLLM, a 1.3B bilingual language model trained on balanced English and French datasets, achieves high performance on consumer hardware and is validated using custom transparency criteria in FrenchBench.
paper_ai_keywords: MLM CroissantLLM pretraining bilingual model custom tokenizer bilingual finetuning FrenchBench model performance FMTI framework transparency criteria multilinguality