title: Value-Based Deep RL Scales Predictably
summary: Scaling data and compute is critical to the success of machine learning.
  However, scaling demands predictability: we want methods to not only perform
  well with more compute or data, but also have their performance be predictable
  from small-scale runs, without running the large-scale experiment. In this
  paper, we show that value-based off-policy RL methods are predictable despite
  community lore regarding their pathological behavior. First, we show that data
  and compute requirements to attain a given performance level lie on a Pareto
  frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
  frontier, we can predict this data requirement when given more compute, and
  this compute requirement when given more data. Second, we determine the optimal
  allocation of a total resource budget across data and compute for a given
  performance and use it to determine hyperparameters that maximize performance
  for a given budget. Third, this scaling behavior is enabled by first estimating
  predictable relationships between hyperparameters, which is used to manage
  effects of overfitting and plasticity loss unique to RL. We validate our
  approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
  gym, and IsaacGym, when extrapolating to higher levels of data, compute,
  budget, or performance.
paper_authors._id: 67aa44a0927861da2b7a3479
paper_authors.name: Oleh Rybkin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-10T18:25:37.833Z
paper_authors._id: 64d1161315b26cc7f70f37e6
paper_authors.avatarUrl: /avatars/c020b7d2c6c2bb1ca289a9cf0c4eaf00.svg
paper_authors.fullname: Oleh Rybkin
paper_authors.type: user
paper_authors.user: orybkin
paper_authors._id: 67aa44a0927861da2b7a347a
paper_authors.name: Michal Nauman
paper_authors._id: 67aa44a0927861da2b7a347b
paper_authors.name: Preston Fu
paper_authors._id: 67aa44a0927861da2b7a347c
paper_authors.name: Charlie Snell
paper_authors._id: 67aa44a0927861da2b7a347d
paper_authors.name: Pieter Abbeel
paper_authors._id: 67aa44a0927861da2b7a347e
paper_authors.name: Sergey Levine
paper_authors._id: 67aa44a0927861da2b7a347f
paper_authors.name: Aviral Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T14:25:38.016Z
paper_authors._id: 67315a324d2eab8035de786a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/INH5_PlcJYfYcGkyibjgP.png
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralku
paper_title: Value-Based Deep RL Scales Predictably
paper_summary: Scaling data and compute is critical to the success of machine learning.
  However, scaling demands predictability: we want methods to not only perform
  well with more compute or data, but also have their performance be predictable
  from small-scale runs, without running the large-scale experiment. In this
  paper, we show that value-based off-policy RL methods are predictable despite
  community lore regarding their pathological behavior. First, we show that data
  and compute requirements to attain a given performance level lie on a Pareto
  frontier, controlled by the updates-to-data (UTD) ratio. By estimating this
  frontier, we can predict this data requirement when given more compute, and
  this compute requirement when given more data. Second, we determine the optimal
  allocation of a total resource budget across data and compute for a given
  performance and use it to determine hyperparameters that maximize performance
  for a given budget. Third, this scaling behavior is enabled by first estimating
  predictable relationships between hyperparameters, which is used to manage
  effects of overfitting and plasticity loss unique to RL. We validate our
  approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI
  gym, and IsaacGym, when extrapolating to higher levels of data, compute,
  budget, or performance.
paper_ai_keywords: off-policy RL value-based methods Pareto frontier updates-to-data (UTD) ratio hyperparameters overfitting plasticity loss SAC BRO PQL DeepMind Control OpenAI gym IsaacGym