title: SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
    Learning
summary: Recent advances in CV and NLP have been largely driven by scaling up the
  number of network parameters, despite traditional theories suggesting that
  larger networks are prone to overfitting. These large networks avoid
  overfitting by integrating components that induce a simplicity bias, guiding
  models toward simple and generalizable solutions. However, in deep RL,
  designing and scaling up networks have been less explored. Motivated by this
  opportunity, we present SimBa, an architecture designed to scale up parameters
  in deep RL by injecting a simplicity bias. SimBa consists of three components:
  (i) an observation normalization layer that standardizes inputs with running
  statistics, (ii) a residual feedforward block to provide a linear pathway from
  the input to output, and (iii) a layer normalization to control feature
  magnitudes. By scaling up parameters with SimBa, the sample efficiency of
  various deep RL algorithms-including off-policy, on-policy, and unsupervised
  methods-is consistently improved. Moreover, solely by integrating SimBa
  architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
  with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
  These results demonstrate SimBa's broad applicability and effectiveness across
  diverse RL algorithms and environments.
paper_authors._id: 670f6e287d1b0373c6419900
paper_authors.name: Hojoon Lee
paper_authors._id: 670f6e287d1b0373c6419901
paper_authors.name: Dongyoon Hwang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:43:08.864Z
paper_authors._id: 67061bf00cc840f4462a0705
paper_authors.avatarUrl: /avatars/0fab53fa981f166483aa8ad61172d32d.svg
paper_authors.fullname: Dongyoon
paper_authors.type: user
paper_authors.user: godnpeter
paper_authors._id: 670f6e287d1b0373c6419902
paper_authors.name: Donghu Kim
paper_authors._id: 670f6e287d1b0373c6419903
paper_authors.name: Hyunseung Kim
paper_authors._id: 670f6e287d1b0373c6419904
paper_authors.name: Jun Jet Tai
paper_authors._id: 670f6e287d1b0373c6419905
paper_authors.name: Kaushik Subramanian
paper_authors._id: 670f6e287d1b0373c6419906
paper_authors.name: Peter R. Wurman
paper_authors._id: 670f6e287d1b0373c6419907
paper_authors.name: Jaegul Choo
paper_authors._id: 670f6e287d1b0373c6419908
paper_authors.name: Peter Stone
paper_authors._id: 670f6e287d1b0373c6419909
paper_authors.name: Takuma Seno
paper_title: SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
    Learning
paper_summary: Recent advances in CV and NLP have been largely driven by scaling up the
  number of network parameters, despite traditional theories suggesting that
  larger networks are prone to overfitting. These large networks avoid
  overfitting by integrating components that induce a simplicity bias, guiding
  models toward simple and generalizable solutions. However, in deep RL,
  designing and scaling up networks have been less explored. Motivated by this
  opportunity, we present SimBa, an architecture designed to scale up parameters
  in deep RL by injecting a simplicity bias. SimBa consists of three components:
  (i) an observation normalization layer that standardizes inputs with running
  statistics, (ii) a residual feedforward block to provide a linear pathway from
  the input to output, and (iii) a layer normalization to control feature
  magnitudes. By scaling up parameters with SimBa, the sample efficiency of
  various deep RL algorithms-including off-policy, on-policy, and unsupervised
  methods-is consistently improved. Moreover, solely by integrating SimBa
  architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
  with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
  These results demonstrate SimBa's broad applicability and effectiveness across
  diverse RL algorithms and environments.
paper_ai_keywords: deep RL SimBa observation normalization layer residual feedforward block layer normalization sample efficiency off-policy on-policy unsupervised methods SAC DMC MyoSuite HumanoidBench