Answers.Body: <h1>Check the function loss.</h1>
  <p>It might be that your environment is impossible to learn. However, most likely the network simply can't handle it. By measuring the loss during the learning stage, if you find it is always very high and does not decrease, it's a strong indication this might be the issue.</p>
  <p>Because the network is too simple, when you optimize for some states, you ruin others. There is not formal way to find out if this is the case, but since the same algorithm works elsewhere, it's either a problem of your environment, or of the network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-03T11:09:55.523Z
Answers.Text: @Gulzar it can. Not by virtue of being unable to learn, but by taking too long to do all the calculations do to its size. Ideally, we want the smallest and fastest network possible that can learn the function we are modelling
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-31T17:05:53.013Z
Answers.LastActivityDate: 2019-01-31T17:05:53.013Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: BlueMoon93
Answers.Body: <p>Loss not decreasing during the training does not signify that network don't train. It may mean network is in the exploration mode. I have observed that situation - accumulated reward growing steadily, which mean network is training well, but loss is not decreasing.</p>
  
  <p>If you already know solution to some (simpler or other) version of your environment, you can train network in supervised manner to reproduce that solution. If network unable reproduce existing solution that is strong indication that network is too small or otherwise not good.</p>
  
  <p>The other reason for accumulated reward oscillation could be network overfitting on latest training samples. In that case bigger replay buffer or more slow update of target network(if target network is used) may help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-03T06:24:59.297Z
Answers.LastActivityDate: 2019-03-03T06:24:59.297Z
Answers.UserDisplayName: mirror2image
Body: <p>I am using <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">Open AI's</a> code to do a RL task on an environment that I built myself.</p>
  
  <p>I tried some network architectures, and they all converge, faster or slower on CartPole.</p>
  
  <p>On my environment, the reward seems not to converge, and keeps flickering forever.</p>
  
  <p>I suspect the neural network is too small, but I want to confirm my belief before going the route of researching the architecture.</p>
  
  <p>How can I confirm that the architecture is the problem and not anything else in a neural network reinforcement learning task?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><reinforcement-learning>
Title: How to identify too small network in reinforcement learning?
UserDisplayName: Gulzar
ThreadText: # How to identify too small network in reinforcement learning?
  
  (Asked by: username_0 on 2019-01-26)
  
  I am using [Open AI's](https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py) code to do a RL task on an environment that I built myself.
  
  I tried some network architectures, and they all converge, faster or slower on CartPole.
  
  On my environment, the reward seems not to converge, and keeps flickering forever.
  
  I suspect the neural network is too small, but I want to confirm my belief before going the route of researching the architecture.
  
  How can I confirm that the architecture is the problem and not anything else in a neural network reinforcement learning task?
  
  **Tags:** neural-networks, machine-learning, deep-learning, reinforcement-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-01-31. Score: 1
  
  # Check the function loss.
  
  It might be that your environment is impossible to learn. However, most likely the network simply can't handle it. By measuring the loss during the learning stage, if you find it is always very high and does not decrease, it's a strong indication this might be the issue.
  
  Because the network is too simple, when you optimize for some states, you ruin others. There is not formal way to find out if this is the case, but since the same algorithm works elsewhere, it's either a problem of your environment, or of the network.
  
  > **Comments:**
  > **username_1**: @Gulzar it can. Not by virtue of being unable to learn, but by taking too long to do all the calculations do to its size. Ideally, we want the smallest and fastest network possible that can learn the function we are modelling  
  
  ---
  
  ### Answer by username_2 on 2019-03-03. Score: 0
  
  Loss not decreasing during the training does not signify that network don't train. It may mean network is in the exploration mode. I have observed that situation - accumulated reward growing steadily, which mean network is training well, but loss is not decreasing.
  
  If you already know solution to some (simpler or other) version of your environment, you can train network in supervised manner to reproduce that solution. If network unable reproduce existing solution that is strong indication that network is too small or otherwise not good.
  
  The other reason for accumulated reward oscillation could be network overfitting on latest training samples. In that case bigger replay buffer or more slow update of target network(if target network is used) may help.
  
  
  ---
  
  