title: NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
summary: Aligning Large Language Models (LLMs) with human values and preferences is
  essential for making them helpful and safe. However, building efficient tools
  to perform alignment can be challenging, especially for the largest and most
  competent LLMs which often contain tens or hundreds of billions of parameters.
  We create NeMo-Aligner, a toolkit for model alignment that can efficiently
  scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
  optimized and scalable implementations for major paradigms of model alignment
  such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
  Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
  our toolkit supports running most of the alignment techniques in a Parameter
  Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
  extensibility, allowing support for other alignment techniques with minimal
  effort. It is open-sourced with Apache 2.0 License and we invite community
  contributions at https://github.com/NVIDIA/NeMo-Aligner
paper_authors._id: 66344c29dcf96f82195c53bd
paper_authors.name: Gerald Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T17:32:26.507Z
paper_authors._id: 6635187e27cf4d1a75f27a40
paper_authors.avatarUrl: /avatars/e36f11c962f52e783b4677ca5460c41e.svg
paper_authors.fullname: Gerald Shen
paper_authors.type: user
paper_authors.user: gshennvm
paper_authors._id: 66344c29dcf96f82195c53be
paper_authors.name: Zhilin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:55:41.461Z
paper_authors._id: 635865c8aff68f72ac039a4e
paper_authors.avatarUrl: /avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
paper_authors.fullname: Zhilin Wang
paper_authors.type: user
paper_authors.user: zhilinw
paper_authors._id: 66344c29dcf96f82195c53bf
paper_authors.name: Olivier Delalleau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:55:49.203Z
paper_authors._id: 6556379e10428134ff235afd
paper_authors.avatarUrl: /avatars/ec569729870d7392e806e59a02f37d0c.svg
paper_authors.fullname: Olivier Delalleau
paper_authors.type: user
paper_authors.user: odelalleau
paper_authors._id: 66344c29dcf96f82195c53c0
paper_authors.name: Jiaqi Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T17:32:41.399Z
paper_authors._id: 6511cb8d254ae1f96174a330
paper_authors.avatarUrl: /avatars/525c88f6c146fa83e0954d8ed97ee99e.svg
paper_authors.fullname: jiaqiz
paper_authors.type: user
paper_authors.user: jiaqiz
paper_authors._id: 66344c29dcf96f82195c53c1
paper_authors.name: Yi Dong
paper_authors._id: 66344c29dcf96f82195c53c2
paper_authors.name: Daniel Egert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:56:24.480Z
paper_authors._id: 63d2f2bc2727d7888cb9f3cd
paper_authors.avatarUrl: /avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
paper_authors.fullname: Daniel Egert
paper_authors.type: user
paper_authors.user: trias702
paper_authors._id: 66344c29dcf96f82195c53c3
paper_authors.name: Shengyang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:56:30.318Z
paper_authors._id: 6627ed332c3877cd9db6502f
paper_authors.avatarUrl: /avatars/fa6d074adf0e116e93592280fc062b4f.svg
paper_authors.fullname: shengyangsun
paper_authors.type: user
paper_authors.user: shengyangsun
paper_authors._id: 66344c29dcf96f82195c53c4
paper_authors.name: Jimmy Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-06T07:02:59.647Z
paper_authors._id: 6556a4fb8076124b8d75d106
paper_authors.avatarUrl: /avatars/0c8a2a53aa952f3e07be6fd758e0ef8a.svg
paper_authors.fullname: Jimmy Zhang
paper_authors.type: user
paper_authors.user: jiemingz
paper_authors._id: 66344c29dcf96f82195c53c5
paper_authors.name: Sahil Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-06T07:03:01.468Z
paper_authors._id: 644c1d5a8c51ddbe0ea6d920
paper_authors.avatarUrl: /avatars/6864599407d9b75e11860e27032b706d.svg
paper_authors.fullname: Sahil Jain
paper_authors.type: user
paper_authors.user: Sahiljain314
paper_authors._id: 66344c29dcf96f82195c53c6
paper_authors.name: Ali Taghibakhshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:57:17.204Z
paper_authors._id: 654a4bd5b9cfada0bd39536e
paper_authors.avatarUrl: /avatars/d7663dd476a4e263ae7a288bb6bd2994.svg
paper_authors.fullname: Ali Taghibakhshi
paper_authors.type: user
paper_authors.user: jrd971000
paper_authors._id: 66344c29dcf96f82195c53c7
paper_authors.name: Markel Sanz Ausin
paper_authors._id: 66344c29dcf96f82195c53c8
paper_authors.name: Ashwath Aithal
paper_authors._id: 66344c29dcf96f82195c53c9
paper_authors.name: Oleksii Kuchaiev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:57:46.240Z
paper_authors._id: 622937a4acd5bef90e55c49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
paper_authors.fullname: Oleksii Kuchaiev
paper_authors.type: user
paper_authors.user: okuchaiev
paper_title: NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
paper_summary: Aligning Large Language Models (LLMs) with human values and preferences is
  essential for making them helpful and safe. However, building efficient tools
  to perform alignment can be challenging, especially for the largest and most
  competent LLMs which often contain tens or hundreds of billions of parameters.
  We create NeMo-Aligner, a toolkit for model alignment that can efficiently
  scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
  optimized and scalable implementations for major paradigms of model alignment
  such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
  Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
  our toolkit supports running most of the alignment techniques in a Parameter
  Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
  extensibility, allowing support for other alignment techniques with minimal
  effort. It is open-sourced with Apache 2.0 License and we invite community
  contributions at https://github.com/NVIDIA/NeMo-Aligner
paper_ai_summary: NeMo-Aligner is a toolkit for aligning large language models with human values using scalable techniques like RLHF, DPO, SteerLM, and SPIN, optimized for use with hundreds of GPUs and parameter-efficient fine-tuning.
paper_ai_keywords: NeMo-Aligner Reinforcement Learning from Human Feedback (RLHF) Direct Preference Optimization (DPO) SteerLM Self-Play Fine-Tuning (SPIN) Parameter Efficient Fine-Tuning (PEFT)