Answers.Body: <p>I'll share my understanding so far.
  This kind of behavior is actually normal when using on-policy algorithms with a sparse final reward.
  Issue stems from the fact that once you get stuck in a behavior policy which does nothing (uses a "do nothing" action, for instance, until timeout), it's quite hard to get out of it, because you keep getting experiences that teach you nothing (no reward signal at all) and keeps you in the same policy.
  Possible mitigations:</p>
  
  <ul>
  <li>Encourage more exploration (in A3C, make the entropy loss coefficient bigger) in order to recover more quickly from this type of stationary behavior.</li>
  <li>Use an off-policy algorithm with a big enough replay buffer, so that even if you start behaving this way you still use experiences from a "healthy" older policy. </li>
  </ul>
  
  <p>If you stick to a totally on-policy algorithm, making the batch_size bigger might help a little.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-06T15:37:27.367Z
Answers.LastActivityDate: 2019-11-06T15:37:27.367Z
Answers.UserDisplayName: Gabizon
Body: <p>The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause.</p>
  
  <p><a href="https://i.sstatic.net/NGQeV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NGQeV.png" alt="enter image description here"></a></p>
  
  <p>The problem has a sparse reward, batch <code>size=2000</code>, <code>entropy_coefficient=0.1</code>, other hyper-parameters are pretty standard.</p>
  
  <p>Has anyone seen this kind of behavior? What could be the cause these drops in the reward(not enough exploration, too sparse rewards, the state not expressive enough, etc.)?</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><rewards><actor-critic-methods>
Title: What could be the cause of the drop in the reward in A3C?
UserDisplayName: Gabizon
ThreadText: # What could be the cause of the drop in the reward in A3C?
  
  (Asked by: username_0 on 2019-10-28)
  
  The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause.
  
  [![enter image description here](https://i.sstatic.net/NGQeV.png)](https://i.sstatic.net/NGQeV.png)
  
  The problem has a sparse reward, batch `size=2000`, `entropy_coefficient=0.1`, other hyper-parameters are pretty standard.
  
  Has anyone seen this kind of behavior? What could be the cause these drops in the reward(not enough exploration, too sparse rewards, the state not expressive enough, etc.)?
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, rewards, actor-critic-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-11-06. Score: 0
  
  I'll share my understanding so far.
  This kind of behavior is actually normal when using on-policy algorithms with a sparse final reward.
  Issue stems from the fact that once you get stuck in a behavior policy which does nothing (uses a "do nothing" action, for instance, until timeout), it's quite hard to get out of it, because you keep getting experiences that teach you nothing (no reward signal at all) and keeps you in the same policy.
  Possible mitigations:
  
  * Encourage more exploration (in A3C, make the entropy loss coefficient bigger) in order to recover more quickly from this type of stationary behavior.
  * Use an off-policy algorithm with a big enough replay buffer, so that even if you start behaving this way you still use experiences from a "healthy" older policy.
  
  If you stick to a totally on-policy algorithm, making the batch\_size bigger might help a little.
  
  
  ---
  
  