Body: <p>First of all I know that: 'it makes training less stable' &amp; 'RL is already inherently unstable'. I'm asking why those things are true?</p>
  <p>Intuitively it seems very strange &amp; to be perhaps a fundamental weakness of RL that it
  very ineffective at learning large networks. Supervised learning usually has no problem learning features from unstructured data via potentially quite large networks (e.g. upwards of 8+ layers). But in the case of RL it is a very big problem it seems (that is learning the extra embedding layers in the front of the network 'head').</p>
  <p>P.S. in case you don't believe me here is a plot of hyper-parameter search data I collected on Acrobot-v1, CartPole, MountainCar-v0, and Pendulum-v0.
  Also it is not hard to find evidence for this online... Just look at stable-baselines3 it has default num_layers=2...</p>
  <p><a href="https://i.sstatic.net/qTRqu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qTRqu.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-18T17:08:43.350Z
Comments.Text: I know that it might be true, but who said that "training large networks in RL is usually (more) unstable"? It might be a good idea to quote someone or something that claimed this, or maybe tell us that you've observed this empirically.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-23T21:36:47.863Z
Comments.Text: Sometimes the initial layers of the model will approximate the environment better. So adding layers means adding more noise and hence the learning becomes unstable. Every neural network has certain optimal number of layers and depth depending upon the use case. Too many layers in a neural network leads to bad learning and it is not limited only to RL. One of the areas I see this more frequently is in image classification where in some cases, few layers perform better than large number of layers
Comments.UserDisplayName: MonkeyDLuffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-24T21:03:35.867Z
Comments.Text: @MonkeyDLuffy I appreciate your answer but this is definitely something special about RL. The 'standard' number of layers is 2! Compared to 8 for even a pretty standard feedforward network, something along these lines is common in computer vision/NLP too.
  
  In RL that leads to learning collapse.
Comments.UserDisplayName: profPlum
Tags: <reinforcement-learning><deep-learning><deep-rl>
Title: Why is it difficult to train large RL networks?
UserDisplayName: profPlum
ThreadText: # Why is it difficult to train large RL networks?
  
  (Asked by: username_0 on 2022-06-18)
  
  First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true?
  
  Intuitively it seems very strange & to be perhaps a fundamental weakness of RL that it
  very ineffective at learning large networks. Supervised learning usually has no problem learning features from unstructured data via potentially quite large networks (e.g. upwards of 8+ layers). But in the case of RL it is a very big problem it seems (that is learning the extra embedding layers in the front of the network 'head').
  
  P.S. in case you don't believe me here is a plot of hyper-parameter search data I collected on Acrobot-v1, CartPole, MountainCar-v0, and Pendulum-v0.
  Also it is not hard to find evidence for this online... Just look at stable-baselines3 it has default num\_layers=2...
  
  [![enter image description here](https://i.sstatic.net/qTRqu.png)](https://i.sstatic.net/qTRqu.png)
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl
  
  > **Comments:**
  > **username_1**: I know that it might be true, but who said that "training large networks in RL is usually (more) unstable"? It might be a good idea to quote someone or something that claimed this, or maybe tell us that you've observed this empirically.  
  > **username_2**: Sometimes the initial layers of the model will approximate the environment better. So adding layers means adding more noise and hence the learning becomes unstable. Every neural network has certain optimal number of layers and depth depending upon the use case. Too many layers in a neural network leads to bad learning and it is not limited only to RL. One of the areas I see this more frequently is in image classification where in some cases, few layers perform better than large number of layers  
  > **username_0**: @username_2 I appreciate your answer but this is definitely something special about RL. The 'standard' number of layers is 2! Compared to 8 for even a pretty standard feedforward network, something along these lines is common in computer vision/NLP too.  In RL that leads to learning collapse.  
  
  ---
  
  ## 0 answers
  
  