title: DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like
    Models at All Scales
summary: ChatGPT-like models have revolutionized various applications in artificial
  intelligence, from summarization and coding to translation, matching or even
  surpassing human performance. However, the current landscape lacks an
  accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement
  Learning with Human Feedback) training pipeline for these powerful models,
  particularly when training at the scale of billions of parameters. This paper
  introduces DeepSpeed-Chat, a novel system that democratizes RLHF training,
  making it accessible to the AI community. DeepSpeed-Chat offers three key
  capabilities: an easy-to-use training and inference experience for ChatGPT-like
  models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from
  InstructGPT, and a robust DeepSpeed-RLHF system that combines various
  optimizations for training and inference in a unified way. The system delivers
  unparalleled efficiency and scalability, enabling training of models with
  hundreds of billions of parameters in record time and at a fraction of the
  cost. With this development, DeepSpeed-Chat paves the way for broader access to
  advanced RLHF training, even for data scientists with limited resources,
  thereby fostering innovation and further development in the field of AI.
paper_authors._id: 64cc542186d8dc0caa684f7f
paper_authors.name: Zhewei Yao
paper_authors._id: 64cc542186d8dc0caa684f80
paper_authors.name: Reza Yazdani Aminabadi
paper_authors._id: 64cc542186d8dc0caa684f81
paper_authors.name: Olatunji Ruwase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:43:15.265Z
paper_authors._id: 6499d685ea2cdac80992e742
paper_authors.avatarUrl: /avatars/c32741e7fc57c9a08722fab3877a7b81.svg
paper_authors.fullname: Olatunji Ruwase
paper_authors.type: user
paper_authors.user: tjruwase
paper_authors._id: 64cc542186d8dc0caa684f82
paper_authors.name: Samyam Rajbhandari
paper_authors._id: 64cc542186d8dc0caa684f83
paper_authors.name: Xiaoxia Wu
paper_authors._id: 64cc542186d8dc0caa684f84
paper_authors.name: Ammar Ahmad Awan
paper_authors._id: 64cc542186d8dc0caa684f85
paper_authors.name: Jeff Rasley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:44:44.408Z
paper_authors._id: 60871172511e863acdb20b9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60871172511e863acdb20b9f/qlhLl8hAryhQY3uvjoZ3i.jpeg
paper_authors.fullname: Jeff Rasley
paper_authors.type: user
paper_authors.user: jeffra
paper_authors._id: 64cc542186d8dc0caa684f86
paper_authors.name: Minjia Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T08:17:09.769Z
paper_authors._id: 6305b5e39d2531fabd195c5f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Minjia
paper_authors._id: 64cc542186d8dc0caa684f87
paper_authors.name: Conglong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:45:12.425Z
paper_authors._id: 617adc6d941993035fbaf2b3
paper_authors.avatarUrl: /avatars/8de7dc3a6e202a49ac9c061e93d0fc92.svg
paper_authors.fullname: Conglong Li
paper_authors.type: user
paper_authors.user: conglongli
paper_authors._id: 64cc542186d8dc0caa684f88
paper_authors.name: Connor Holmes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:45:37.842Z
paper_authors._id: 630e826cc0eca3037a04e9c7
paper_authors.avatarUrl: /avatars/d71f42635f3e678acae4eb99cf9d8431.svg
paper_authors.fullname: Connor Holmes
paper_authors.type: user
paper_authors.user: cmikeh2
paper_authors._id: 64cc542186d8dc0caa684f89
paper_authors.name: Zhongzhu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:46:17.539Z
paper_authors._id: 6426781213a9e5d9675a1d57
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vR7u0azaShh5Tj_OqdP2k.png
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Zhongzhu
paper_authors._id: 64cc542186d8dc0caa684f8a
paper_authors.name: Michael Wyatt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:47:01.735Z
paper_authors._id: 634efda78d0f051a4506811a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666121085635-noauth.png
paper_authors.fullname: Michael Wyatt Miller
paper_authors.type: user
paper_authors.user: michAAAAELLL
paper_authors._id: 64cc542186d8dc0caa684f8b
paper_authors.name: Molly Smith
paper_authors._id: 64cc542186d8dc0caa684f8c
paper_authors.name: Lev Kurilenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:47:44.419Z
paper_authors._id: 635852141d66b442317cbe26
paper_authors.avatarUrl: /avatars/bcf9164f6ac6393ffecc5f4383d12cbe.svg
paper_authors.fullname: Lev Kurilenko
paper_authors.type: user
paper_authors.user: lekurile
paper_authors._id: 64cc542186d8dc0caa684f8d
paper_authors.name: Heyang Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T11:47:59.361Z
paper_authors._id: 63bc6a611374e3ef9134a24e
paper_authors.avatarUrl: /avatars/9df037b9cf7d7952a10d919e24a2ffa2.svg
paper_authors.fullname: Heyang Qin
paper_authors.type: user
paper_authors.user: heyangqin
paper_authors._id: 64cc542186d8dc0caa684f8e
paper_authors.name: Masahiro Tanaka
paper_authors._id: 64cc542186d8dc0caa684f8f
paper_authors.name: Shuai Che
paper_authors._id: 64cc542186d8dc0caa684f90
paper_authors.name: Shuaiwen Leon Song
paper_authors._id: 64cc542186d8dc0caa684f91
paper_authors.name: Yuxiong He
paper_title: DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like
    Models at All Scales
paper_summary: ChatGPT-like models have revolutionized various applications in artificial
  intelligence, from summarization and coding to translation, matching or even
  surpassing human performance. However, the current landscape lacks an
  accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement
  Learning with Human Feedback) training pipeline for these powerful models,
  particularly when training at the scale of billions of parameters. This paper
  introduces DeepSpeed-Chat, a novel system that democratizes RLHF training,
  making it accessible to the AI community. DeepSpeed-Chat offers three key
  capabilities: an easy-to-use training and inference experience for ChatGPT-like
  models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from
  InstructGPT, and a robust DeepSpeed-RLHF system that combines various
  optimizations for training and inference in a unified way. The system delivers
  unparalleled efficiency and scalability, enabling training of models with
  hundreds of billions of parameters in record time and at a fraction of the
  cost. With this development, DeepSpeed-Chat paves the way for broader access to
  advanced RLHF training, even for data scientists with limited resources,
  thereby fostering innovation and further development in the field of AI.
paper_ai_keywords: RLHF Reinforcement Learning with Human Feedback DeepSpeed-Chat DeepSpeed-RLHF InstructGPT