title: Mixtures of Experts Unlock Parameter Scaling for Deep RL
summary: The recent rapid progress in (self) supervised learning models is in large
  part predicted by empirical scaling laws: a model's performance scales
  proportionally to its size. Analogous scaling laws remain elusive for
  reinforcement learning domains, however, where increasing the parameter count
  of a model often hurts its final performance. In this paper, we demonstrate
  that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
  (Puigcerver et al., 2023), into value-based networks results in more
  parameter-scalable models, evidenced by substantial performance increases
  across a variety of training regimes and model sizes. This work thus provides
  strong empirical evidence towards developing scaling laws for reinforcement
  learning.
paper_authors._id: 65cc30b751c1738a55e94ba9
paper_authors.name: Johan Obando-Ceron
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:41:41.148Z
paper_authors._id: 6478f30ea68454566353ef95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6478f30ea68454566353ef95/hX5NiVPyQbK8TIBKnT4J1.jpeg
paper_authors.fullname: Johan Samir Obando Ceron
paper_authors.type: user
paper_authors.user: johanobandoc
paper_authors._id: 65cc30b751c1738a55e94baa
paper_authors.name: Ghada Sokar
paper_authors._id: 65cc30b751c1738a55e94bab
paper_authors.name: Timon Willi
paper_authors._id: 65cc30b751c1738a55e94bac
paper_authors.name: Clare Lyle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T10:41:45.967Z
paper_authors._id: 6313f69d54e6e5d9f0fb9f10
paper_authors.avatarUrl: /avatars/12114d479bb6dc394d29f988944f6d47.svg
paper_authors.fullname: Clare Lyle
paper_authors.type: user
paper_authors.user: justclarifying
paper_authors._id: 65cc30b751c1738a55e94bad
paper_authors.name: Jesse Farebrother
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T10:42:01.932Z
paper_authors._id: 64b85c41cf14c2fabe8edd37
paper_authors.avatarUrl: /avatars/900d99b3b24aac5c98fb1f50eef8e825.svg
paper_authors.fullname: Jesse Farebrother
paper_authors.type: user
paper_authors.user: jessefarebro
paper_authors._id: 65cc30b751c1738a55e94bae
paper_authors.name: Jakob Foerster
paper_authors._id: 65cc30b751c1738a55e94baf
paper_authors.name: Gintare Karolina Dziugaite
paper_authors._id: 65cc30b751c1738a55e94bb0
paper_authors.name: Doina Precup
paper_authors._id: 65cc30b751c1738a55e94bb1
paper_authors.name: Pablo Samuel Castro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T10:42:53.652Z
paper_authors._id: 6303fabc7373aacccd87ddcf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661205153867-noauth.png
paper_authors.fullname: Pablo Samuel Castro
paper_authors.type: user
paper_authors.user: pcastr
paper_title: Mixtures of Experts Unlock Parameter Scaling for Deep RL
paper_summary: The recent rapid progress in (self) supervised learning models is in large
  part predicted by empirical scaling laws: a model's performance scales
  proportionally to its size. Analogous scaling laws remain elusive for
  reinforcement learning domains, however, where increasing the parameter count
  of a model often hurts its final performance. In this paper, we demonstrate
  that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
  (Puigcerver et al., 2023), into value-based networks results in more
  parameter-scalable models, evidenced by substantial performance increases
  across a variety of training regimes and model sizes. This work thus provides
  strong empirical evidence towards developing scaling laws for reinforcement
  learning.
paper_ai_keywords: self-supervised learning empirical scaling laws Mixture-of-Expert (MoE) modules Soft MoEs value-based networks parameter-scalable models