title: TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
summary: Language model post-training is applied to refine behaviors and unlock new
  skills across a wide range of recent language models, but open recipes for
  applying these techniques lag behind proprietary ones. The underlying training
  data and recipes for post-training are simultaneously the most important pieces
  of the puzzle and the portion with the least transparency. To bridge this gap,
  we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained
  models, alongside its data, code, and training recipes, serving as a
  comprehensive guide for modern post-training techniques. T\"ULU 3, which builds
  on Llama 3.1 base models, achieves results surpassing the instruct versions of
  Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
  Claude 3.5-Haiku. The training algorithms for our models include supervised
  finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
  call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we
  introduce a multi-task evaluation scheme for post-training recipes with
  development and unseen evaluations, standard benchmark implementations, and
  substantial decontamination of existing open datasets on said benchmarks. We
  conclude with analysis and discussion of training methods that did not reliably
  improve performance.
    In addition to the T\"ULU 3 model weights and demo, we release the complete
  recipe -- including datasets for diverse core skills, a robust toolkit for data
  curation and evaluation, the training code and infrastructure, and, most
  importantly, a detailed report for reproducing and further adapting the T\"ULU
  3 approach to more domains.
paper_authors._id: 674414c8d3ad4510c1a26134
paper_authors.name: Nathan Lambert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T20:52:52.869Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 674414c8d3ad4510c1a26135
paper_authors.name: Jacob Morrison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:40:52.602Z
paper_authors._id: 63f51f6cb51da4d61da72937
paper_authors.avatarUrl: /avatars/6023ecaafb28b7599f59816381d58bd0.svg
paper_authors.fullname: Jacob Morrison
paper_authors.type: user
paper_authors.user: jacobmorrison
paper_authors._id: 674414c8d3ad4510c1a26136
paper_authors.name: Valentina Pyatkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:40:58.240Z
paper_authors._id: 6556cff80e7a7067a934445f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
paper_authors.fullname: Valentina Pyatkin
paper_authors.type: user
paper_authors.user: valpy
paper_authors._id: 674414c8d3ad4510c1a26137
paper_authors.name: Shengyi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:13.686Z
paper_authors._id: 62bfc80b54db40ca617ce53a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bfc80b54db40ca617ce53a/IZxafMqoYJNHwTuFt8AqF.png
paper_authors.fullname: Shengyi Costa Huang
paper_authors.type: user
paper_authors.user: vwxyzjn
paper_authors._id: 674414c8d3ad4510c1a26138
paper_authors.name: Hamish Ivison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:41:19.991Z
paper_authors._id: 62608fc2ffe8827cb1d89f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
paper_authors.fullname: Hamish Ivison
paper_authors.type: user
paper_authors.user: hamishivi
paper_authors._id: 674414c8d3ad4510c1a26139
paper_authors.name: Faeze Brahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:41:25.755Z
paper_authors._id: 65282b8d578679aac7888aec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
paper_authors.fullname: Faeze Brahman
paper_authors.type: user
paper_authors.user: faezeb
paper_authors._id: 674414c8d3ad4510c1a2613a
paper_authors.name: Lester James V. Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:20.277Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 674414c8d3ad4510c1a2613b
paper_authors.name: Alisa Liu
paper_authors._id: 674414c8d3ad4510c1a2613c
paper_authors.name: Nouha Dziri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:41:51.152Z
paper_authors._id: 625888c2c964680d85d70bff
paper_authors.avatarUrl: /avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhadziri
paper_authors._id: 674414c8d3ad4510c1a2613d
paper_authors.name: Shane Lyu
paper_authors._id: 674414c8d3ad4510c1a2613e
paper_authors.name: Yuling Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:05.890Z
paper_authors._id: 62a8b91c1277a49db0c5c080
paper_authors.avatarUrl: /avatars/9cf834a02b67af862659c9b9d4367a52.svg
paper_authors.fullname: Yuling Gu
paper_authors.type: user
paper_authors.user: SaveBertAndGpt
paper_authors._id: 674414c8d3ad4510c1a2613f
paper_authors.name: Saumya Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:11.222Z
paper_authors._id: 63fee2851193ba2c22a1b687
paper_authors.avatarUrl: /avatars/d2357740e01dafb216db0ed2b74a8100.svg
paper_authors.fullname: Saumya Malik
paper_authors.type: user
paper_authors.user: saumyamalik
paper_authors._id: 674414c8d3ad4510c1a26140
paper_authors.name: Victoria Graf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:17.089Z
paper_authors._id: 646d699ed720ccd8cae5c011
paper_authors.avatarUrl: /avatars/2e5ccee268081b7ba663eb6932552345.svg
paper_authors.fullname: Victoria Graf
paper_authors.type: user
paper_authors.user: VGraf
paper_authors._id: 674414c8d3ad4510c1a26141
paper_authors.name: Jena D. Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:25.494Z
paper_authors._id: 65c157eaf9957360824b0d87
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c157eaf9957360824b0d87/c1NWTKsFLATJLR7W6Kmqz.jpeg
paper_authors.fullname: jena hwang
paper_authors.type: user
paper_authors.user: jenah
paper_authors._id: 674414c8d3ad4510c1a26142
paper_authors.name: Jiangjiang Yang
paper_authors._id: 674414c8d3ad4510c1a26143
paper_authors.name: Ronan Le Bras
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:39.456Z
paper_authors._id: 635049104e753c9940fefd71
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
paper_authors.fullname: Ronan Le Bras
paper_authors.type: user
paper_authors.user: ronanlb
paper_authors._id: 674414c8d3ad4510c1a26144
paper_authors.name: Oyvind Tafjord
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:44.624Z
paper_authors._id: 609b1aeff7c1303cfd1760f3
paper_authors.avatarUrl: /avatars/3cb448514885656466886d7c84e12af1.svg
paper_authors.fullname: Oyvind Tafjord
paper_authors.type: user
paper_authors.user: OyvindTafjord
paper_authors._id: 674414c8d3ad4510c1a26145
paper_authors.name: Chris Wilhelm
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:51.192Z
paper_authors._id: 62bf74ca34afe69e1a1f98e3
paper_authors.avatarUrl: /avatars/75aa4e89ea19a971300a57a208c686a6.svg
paper_authors.fullname: Chris Wilhelm
paper_authors.type: user
paper_authors.user: cmwilhelm
paper_authors._id: 674414c8d3ad4510c1a26146
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:42:58.621Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 674414c8d3ad4510c1a26147
paper_authors.name: Noah A. Smith
paper_authors._id: 674414c8d3ad4510c1a26148
paper_authors.name: Yizhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:43:10.147Z
paper_authors._id: 6269be67d1ac0cde592aba29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
paper_authors.fullname: Yizhong Wang
paper_authors.type: user
paper_authors.user: yizhongw
paper_authors._id: 674414c8d3ad4510c1a26149
paper_authors.name: Pradeep Dasigi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:43:15.425Z
paper_authors._id: 6408fcc93461c51cf735a61e
paper_authors.avatarUrl: /avatars/619f3653911d111f046a5a6c30fc8319.svg
paper_authors.fullname: Pradeep Dasigi
paper_authors.type: user
paper_authors.user: pradeepd
paper_authors._id: 674414c8d3ad4510c1a2614a
paper_authors.name: Hannaneh Hajishirzi
paper_title: TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
paper_summary: Language model post-training is applied to refine behaviors and unlock new
  skills across a wide range of recent language models, but open recipes for
  applying these techniques lag behind proprietary ones. The underlying training
  data and recipes for post-training are simultaneously the most important pieces
  of the puzzle and the portion with the least transparency. To bridge this gap,
  we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained
  models, alongside its data, code, and training recipes, serving as a
  comprehensive guide for modern post-training techniques. T\"ULU 3, which builds
  on Llama 3.1 base models, achieves results surpassing the instruct versions of
  Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
  Claude 3.5-Haiku. The training algorithms for our models include supervised
  finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
  call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we
  introduce a multi-task evaluation scheme for post-training recipes with
  development and unseen evaluations, standard benchmark implementations, and
  substantial decontamination of existing open datasets on said benchmarks. We
  conclude with analysis and discussion of training methods that did not reliably
  improve performance.
    In addition to the T\"ULU 3 model weights and demo, we release the complete
  recipe -- including datasets for diverse core skills, a robust toolkit for data
  curation and evaluation, the training code and infrastructure, and, most
  importantly, a detailed report for reproducing and further adapting the T\"ULU
  3 approach to more domains.
paper_ai_keywords: supervised finetuning (SFT) Direct Preference Optimization (DPO) Reinforcement Learning with Verifiable Rewards (RLVR) multi-task evaluation standard benchmark implementations decontamination