NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Aligning Large Language Models (LLMs) with human values and preferences is
essential for making them helpful and safe. However, building efficient tools
to perform alignment can be challenging, especially for the largest and most
competent LLMs which often contain tens or hundreds of billions of parameters.
We create NeMo-Aligner, a toolkit for model alignment that can efficiently
scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
optimized and scalable implementations for major paradigms of model alignment
such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
our toolkit supports running most of the alignment techniques in a Parameter
Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
extensibility, allowing support for other alignment techniques with minimal
effort. It is open-sourced with Apache 2.0 License and we invite community
contributions at https://github.com/NVIDIA/NeMo-Aligner
66344c29dcf96f82195c53bd
Gerald Shen
admin_assigned
2024-05-03T17:32:26.507Z
6635187e27cf4d1a75f27a40
/avatars/e36f11c962f52e783b4677ca5460c41e.svg
Gerald Shen
user
gshennvm
66344c29dcf96f82195c53be
Zhilin Wang
admin_assigned
2024-05-03T07:55:41.461Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
66344c29dcf96f82195c53bf
Olivier Delalleau
admin_assigned
2024-05-03T07:55:49.203Z
6556379e10428134ff235afd
/avatars/ec569729870d7392e806e59a02f37d0c.svg
Olivier Delalleau
user
odelalleau
66344c29dcf96f82195c53c0
Jiaqi Zeng
claimed_verified
2024-05-03T17:32:41.399Z
6511cb8d254ae1f96174a330
/avatars/525c88f6c146fa83e0954d8ed97ee99e.svg
jiaqiz
user
jiaqiz
66344c29dcf96f82195c53c1
Yi Dong
66344c29dcf96f82195c53c2
Daniel Egert
admin_assigned
2024-05-03T07:56:24.480Z
63d2f2bc2727d7888cb9f3cd
/avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
Daniel Egert
user
trias702
66344c29dcf96f82195c53c3
Shengyang Sun
admin_assigned
2024-05-03T07:56:30.318Z
6627ed332c3877cd9db6502f
/avatars/fa6d074adf0e116e93592280fc062b4f.svg
shengyangsun
user
shengyangsun
66344c29dcf96f82195c53c4
Jimmy Zhang
claimed_verified
2024-05-06T07:02:59.647Z
6556a4fb8076124b8d75d106
/avatars/0c8a2a53aa952f3e07be6fd758e0ef8a.svg
Jimmy Zhang
user
jiemingz
66344c29dcf96f82195c53c5
Sahil Jain
claimed_verified
2024-05-06T07:03:01.468Z
644c1d5a8c51ddbe0ea6d920
/avatars/6864599407d9b75e11860e27032b706d.svg
Sahil Jain
user
Sahiljain314
66344c29dcf96f82195c53c6
Ali Taghibakhshi
admin_assigned
2024-05-03T07:57:17.204Z
654a4bd5b9cfada0bd39536e
/avatars/d7663dd476a4e263ae7a288bb6bd2994.svg
Ali Taghibakhshi
user
jrd971000
66344c29dcf96f82195c53c7
Markel Sanz Ausin
66344c29dcf96f82195c53c8
Ashwath Aithal
66344c29dcf96f82195c53c9
Oleksii Kuchaiev
admin_assigned
2024-05-03T07:57:46.240Z
622937a4acd5bef90e55c49d
https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
Oleksii Kuchaiev
user
okuchaiev
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Aligning Large Language Models (LLMs) with human values and preferences is
essential for making them helpful and safe. However, building efficient tools
to perform alignment can be challenging, especially for the largest and most
competent LLMs which often contain tens or hundreds of billions of parameters.
We create NeMo-Aligner, a toolkit for model alignment that can efficiently
scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
optimized and scalable implementations for major paradigms of model alignment
such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
our toolkit supports running most of the alignment techniques in a Parameter
Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
extensibility, allowing support for other alignment techniques with minimal
effort. It is open-sourced with Apache 2.0 License and we invite community
contributions at https://github.com/NVIDIA/NeMo-Aligner
NeMo-Aligner is a toolkit for aligning large language models with human values using scalable techniques like RLHF, DPO, SteerLM, and SPIN, optimized for use with hundreds of GPUs and parameter-efficient fine-tuning.
NeMo-Aligner Reinforcement Learning from Human Feedback (RLHF) Direct Preference Optimization (DPO) SteerLM Self-Play Fine-Tuning (SPIN) Parameter Efficient Fine-Tuning (PEFT)