title: Stop Regressing: Training Value Functions via Classification for
    Scalable Deep RL
summary: Value functions are a central component of deep reinforcement learning (RL).
  These functions, parameterized by neural networks, are trained using a mean
  squared error regression objective to match bootstrapped target values.
  However, scaling value-based RL methods that use regression to large networks,
  such as high-capacity Transformers, has proven challenging. This difficulty is
  in stark contrast to supervised learning: by leveraging a cross-entropy
  classification loss, supervised methods have scaled reliably to massive
  networks. Observing this discrepancy, in this paper, we investigate whether the
  scalability of deep RL can also be improved simply by using classification in
  place of regression for training value functions. We demonstrate that value
  functions trained with categorical cross-entropy significantly improves
  performance and scalability in a variety of domains. These include: single-task
  RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale
  ResNets, robotic manipulation with Q-transformers, playing Chess without
  search, and a language-agent Wordle task with high-capacity Transformers,
  achieving state-of-the-art results on these domains. Through careful analysis,
  we show that the benefits of categorical cross-entropy primarily stem from its
  ability to mitigate issues inherent to value-based RL, such as noisy targets
  and non-stationarity. Overall, we argue that a simple shift to training value
  functions with categorical cross-entropy can yield substantial improvements in
  the scalability of deep RL at little-to-no cost.
paper_authors._id: 65e95673f42775fd671b4f54
paper_authors.name: Jesse Farebrother
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:13:35.336Z
paper_authors._id: 64b85c41cf14c2fabe8edd37
paper_authors.avatarUrl: /avatars/900d99b3b24aac5c98fb1f50eef8e825.svg
paper_authors.fullname: Jesse Farebrother
paper_authors.type: user
paper_authors.user: jessefarebro
paper_authors._id: 65e95673f42775fd671b4f55
paper_authors.name: Jordi Orbay
paper_authors._id: 65e95673f42775fd671b4f56
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:07:58.506Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: kabut1211
paper_authors._id: 65e95673f42775fd671b4f57
paper_authors.name: Adrien Ali Taïga
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:08:08.469Z
paper_authors._id: 63694e1dbf21b20c568d9bb9
paper_authors.avatarUrl: /avatars/9ed8c8a76ae60de26dede86d298bd64f.svg
paper_authors.fullname: Adrien Ali Taiga
paper_authors.type: user
paper_authors.user: aalitaiga
paper_authors._id: 65e95673f42775fd671b4f58
paper_authors.name: Yevgen Chebotar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:08:17.760Z
paper_authors._id: 64c93f9c9471c54a9438162c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
paper_authors.fullname: Yevgen Chebotar
paper_authors.type: user
paper_authors.user: ychebotar
paper_authors._id: 65e95673f42775fd671b4f59
paper_authors.name: Ted Xiao
paper_authors._id: 65e95673f42775fd671b4f5a
paper_authors.name: Alex Irpan
paper_authors._id: 65e95673f42775fd671b4f5b
paper_authors.name: Sergey Levine
paper_authors._id: 65e95673f42775fd671b4f5c
paper_authors.name: Pablo Samuel Castro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:09:03.406Z
paper_authors._id: 6303fabc7373aacccd87ddcf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661205153867-noauth.png
paper_authors.fullname: Pablo Samuel Castro
paper_authors.type: user
paper_authors.user: pcastr
paper_authors._id: 65e95673f42775fd671b4f5d
paper_authors.name: Aleksandra Faust
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:09:10.317Z
paper_authors._id: 64a3b828225b2121f6b3beb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a3b828225b2121f6b3beb3/44AgAQENc7cJMOWcoK4g_.jpeg
paper_authors.fullname: Aleksandra Faust
paper_authors.type: user
paper_authors.user: sandraorion
paper_authors._id: 65e95673f42775fd671b4f5e
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T10:10:04.363Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_authors._id: 65e95673f42775fd671b4f5f
paper_authors.name: Rishabh Agarwal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-07T05:53:55.930Z
paper_authors._id: 647875e37b370854241b89a8
paper_authors.avatarUrl: /avatars/fd25f7e5960e4e132b9ee689f114820b.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: agarwl
paper_title: Stop Regressing: Training Value Functions via Classification for
    Scalable Deep RL
paper_summary: Value functions are a central component of deep reinforcement learning (RL).
  These functions, parameterized by neural networks, are trained using a mean
  squared error regression objective to match bootstrapped target values.
  However, scaling value-based RL methods that use regression to large networks,
  such as high-capacity Transformers, has proven challenging. This difficulty is
  in stark contrast to supervised learning: by leveraging a cross-entropy
  classification loss, supervised methods have scaled reliably to massive
  networks. Observing this discrepancy, in this paper, we investigate whether the
  scalability of deep RL can also be improved simply by using classification in
  place of regression for training value functions. We demonstrate that value
  functions trained with categorical cross-entropy significantly improves
  performance and scalability in a variety of domains. These include: single-task
  RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale
  ResNets, robotic manipulation with Q-transformers, playing Chess without
  search, and a language-agent Wordle task with high-capacity Transformers,
  achieving state-of-the-art results on these domains. Through careful analysis,
  we show that the benefits of categorical cross-entropy primarily stem from its
  ability to mitigate issues inherent to value-based RL, such as noisy targets
  and non-stationarity. Overall, we argue that a simple shift to training value
  functions with categorical cross-entropy can yield substantial improvements in
  the scalability of deep RL at little-to-no cost.
paper_ai_keywords: deep reinforcement learning (RL) value functions neural networks mean squared error regression bootstrapped target values Transformers regression supervised learning cross-entropy classification SoftMoEs ResNets Q-transformers robotic manipulation Chess Wordle categorical cross-entropy noisy targets non-stationarity