Answers.Body: <p>Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks</p>
  <hr />
  <p><a href="https://arxiv.org/pdf/1703.02660.pdf" rel="nofollow noreferrer">Towards Generalization and Simplicity in Continous Control</a> tries to answer the question &quot;What are the simplest set of ingredients needed to succeed in some of the popular benchmarks?&quot;</p>
  <p>Among them is the representation capacity required of the function approximators.</p>
  <p>It shows that in popular OpenAI gym locomotion benchmarks (Swimmer 3D,  3D Humanoid, Ant, Walker), using <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/linear.html" rel="nofollow noreferrer">Linear</a> and <a href="https://en.wikipedia.org/wiki/Radial_basis_function_kernel" rel="nofollow noreferrer">RBF</a> policy architectures is just as competitive in the reward attained as using neural nets. Here's a summary:</p>
  <ul>
  <li>The more Fourier Featurers RBF has, the better the performance</li>
  <li>A 2 layer neural net policy's performance is slightly short of a 500-feature RBF</li>
  <li>A linear representation is unable to learn some of the locomotion behaviour, e.g walking</li>
  </ul>
  <p>I assume RBF, and its use of Conjugate Gradients, has some shortcomings which make shallow Neural networks more preferrable. (I'm not familiar with RBF, and stand to be corrected).</p>
  <p>Overall, the the authors use these result to suggest that complex policy architectures should not be a default choice in RL unless side-by-side comparisons with simpler alternatives suggest otherwise.</p>
  <p>This might explain the prevalence of shallow networks. It's also possible that since most RL research (presently) doesn't try to show how we can learn better representations using complex policy architectures, but rather how to better make use of representation capacities by addressing issues like sample efficiency, shallow networks are expressive enough in the adapted benchmarks. As expressed in Brail's comment, this is also useful for setting up fair comparisons.</p>
  <p>However, the effect of network architecture design in RL has not been widely pursued.</p>
  <p>It's also worth mentioning the findings of this work <a href="https://arxiv.org/abs/2010.09163" rel="nofollow noreferrer">D2RL, (2020)</a> which shows one can gain significant performance improvement by using input concatenations in the policy and value function, and that more layers in a <em>vanilla</em> neural net decreases the agent performance.</p>
  <p><a href="https://i.sstatic.net/5DtMh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5DtMh.png" alt="Effect of no. of layers on SAC performance" /></a></p>
  <p><em>The effect of increasing the number of FCs in SAC. Performance drops after 2 layers (D2RL)</em></p>
  <p><strong>Further Reads</strong>:</p>
  <p><a href="https://www.reddit.com/r/reinforcementlearning/comments/drph2m/why_is_everyone_using_shallow_neural_networks/" rel="nofollow noreferrer">Why is everyone using shallow networks</a> (Reddit thread)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-23T08:24:46.467Z
Answers.Text: I've keyed in RBF in Google search. Says it's short for "Resting Bitch Face" :-)
Answers.UserDisplayName: mugoh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-23T08:23:32.927Z
Answers.LastActivityDate: 2020-11-23T08:28:48.297Z
Answers.LastEditDate: 2020-11-23T08:28:48.297Z
Answers.UserDisplayName: mugoh
Body: <p>In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. <a href="https://arxiv.org/pdf/1711.06782" rel="nofollow noreferrer">Learning to Reset</a>, <a href="https://arxiv.org/pdf/1705.10528" rel="nofollow noreferrer">Constrained Policy Optimization</a>, <a href="https://arxiv.org/pdf/1705.08551" rel="nofollow noreferrer">Model-based RL with stability guarantees</a> just to name a few - these are papers I personally remember but there are definitely many others.</p>
  <p>I came across <a href="https://ai.stackexchange.com/questions/11539/is-reinforcement-learning-using-shallow-neural-networks-still-deep-reinforcement">this question</a> whose answers generally agree that yes, RL using a shallow network is considered deep RL, but the reason for preference of shallow networks was not part of the question.</p>
  <p>In <a href="https://arxiv.org/pdf/1810.05017.pdf" rel="nofollow noreferrer">MetaMimic</a> (2018), the authors trained the largest neural net RL algorithm at the time (a residual net with 20 convolution layers) for one-shot imitation learning. The paper demonstrates that larger networks as policy approximators generalize better and represent many behaviours.</p>
  <p>So, why are shallow 2 layer networks so widely used?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-22T10:26:30.670Z
Comments.Text: Because the point of the paper is to show performance and efficiency of the proposed algorithm not the approximation capability of neural network. It's only fair to compare algorithms with similar neural network structures and not "artificially" improve results with better network architectures
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-22T11:22:53.653Z
Comments.Text: Thanks @Brale. When doing fair baseline comparisons, similar net structures are necessary. Apart from comparisons, there's generally no other reason to using shallow networks?
Comments.UserDisplayName: mugoh
Tags: <neural-networks><reinforcement-learning><deep-learning><deep-rl>
Title: Why are shallow networks so prevalent in RL?
UserDisplayName: mugoh
ThreadText: # Why are shallow networks so prevalent in RL?
  
  (Asked by: username_0 on 2020-11-22)
  
  In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. [Learning to Reset](https://arxiv.org/pdf/1711.06782), [Constrained Policy Optimization](https://arxiv.org/pdf/1705.10528), [Model-based RL with stability guarantees](https://arxiv.org/pdf/1705.08551) just to name a few - these are papers I personally remember but there are definitely many others.
  
  I came across [this question](https://ai.stackexchange.com/questions/11539/is-reinforcement-learning-using-shallow-neural-networks-still-deep-reinforcement) whose answers generally agree that yes, RL using a shallow network is considered deep RL, but the reason for preference of shallow networks was not part of the question.
  
  In [MetaMimic](https://arxiv.org/pdf/1810.05017.pdf) (2018), the authors trained the largest neural net RL algorithm at the time (a residual net with 20 convolution layers) for one-shot imitation learning. The paper demonstrates that larger networks as policy approximators generalize better and represent many behaviours.
  
  So, why are shallow 2 layer networks so widely used?
  
  **Tags:** neural-networks, reinforcement-learning, deep-learning, deep-rl
  > **Comments:**
  > **username_1**: Because the point of the paper is to show performance and efficiency of the proposed algorithm not the approximation capability of neural network. It's only fair to compare algorithms with similar neural network structures and not "artificially" improve results with better network architectures  
  > **username_0**: Thanks @Brale. When doing fair baseline comparisons, similar net structures are necessary. Apart from comparisons, there's generally no other reason to using shallow networks?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-11-23. Score: 1
  
  Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks
  
  ---
  
  [Towards Generalization and Simplicity in Continous Control](https://arxiv.org/pdf/1703.02660.pdf) tries to answer the question "What are the simplest set of ingredients needed to succeed in some of the popular benchmarks?"
  
  Among them is the representation capacity required of the function approximators.
  
  It shows that in popular OpenAI gym locomotion benchmarks (Swimmer 3D, 3D Humanoid, Ant, Walker), using [Linear](http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/linear.html) and [RBF](https://en.wikipedia.org/wiki/Radial_basis_function_kernel) policy architectures is just as competitive in the reward attained as using neural nets. Here's a summary:
  
  * The more Fourier Featurers RBF has, the better the performance
  * A 2 layer neural net policy's performance is slightly short of a 500-feature RBF
  * A linear representation is unable to learn some of the locomotion behaviour, e.g walking
  
  I assume RBF, and its use of Conjugate Gradients, has some shortcomings which make shallow Neural networks more preferrable. (I'm not familiar with RBF, and stand to be corrected).
  
  Overall, the the authors use these result to suggest that complex policy architectures should not be a default choice in RL unless side-by-side comparisons with simpler alternatives suggest otherwise.
  
  This might explain the prevalence of shallow networks. It's also possible that since most RL research (presently) doesn't try to show how we can learn better representations using complex policy architectures, but rather how to better make use of representation capacities by addressing issues like sample efficiency, shallow networks are expressive enough in the adapted benchmarks. As expressed in Brail's comment, this is also useful for setting up fair comparisons.
  
  However, the effect of network architecture design in RL has not been widely pursued.
  
  It's also worth mentioning the findings of this work [D2RL, (2020)](https://arxiv.org/abs/2010.09163) which shows one can gain significant performance improvement by using input concatenations in the policy and value function, and that more layers in a *vanilla* neural net decreases the agent performance.
  
  [![Effect of no. of layers on SAC performance](https://i.sstatic.net/5DtMh.png)](https://i.sstatic.net/5DtMh.png)
  
  *The effect of increasing the number of FCs in SAC. Performance drops after 2 layers (D2RL)*
  
  **Further Reads**:
  
  [Why is everyone using shallow networks](https://www.reddit.com/r/reinforcementlearning/comments/drph2m/why_is_everyone_using_shallow_neural_networks/) (Reddit thread)
  
  > **Comments:**
  > **username_0**: I've keyed in RBF in Google search. Says it's short for "Resting Bitch Face" :-)  
  
  ---
  
  