Answers.Body: <p>i've seen this in learning using deep Q network. some tips may help you came over this problem :</p>
  <ul>
  <li>use some remembering mechanisms like replay buffer. some times agent forget what it has been learnt. reply buffer remembers the agent what he saw at several episodes ago.</li>
  <li>something else that worked for me was changing the optimizer. As DQN article says , using RMSprop is very useful for learning neural network based agents.</li>
  </ul>
  <p>if these tips dont help you, please give more information about your agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-25T20:34:20.947Z
Answers.Text: thank you. Indeed trying DQN instead of A2C seems to help. If I use a large replay buffer with DQN I don't get the resets, while with a smaller replay buffer I still get resets.
  Could this effect also be related to the very large variance I seem to get in episode rewards? (the individual points in the plot have a wide distribution along they y axis)
Answers.UserDisplayName: qwertz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T08:14:08.243Z
Answers.Text: this question needs more investigate on the network. But maybe it's right because input-output normalization of a network may affect the network. ‌Besides changing the optimizer may help you too. I suggest test RMSprop too. My intuition says me that RMSprop have too flexibility at the begging and then becomes more stable at the end of learning. this helps RL agent to not forget the first experiences.
Answers.UserDisplayName: reza karbasi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T15:59:38.503Z
Answers.Text: A2C is on-policy,  so cannot use a long term replay table. It is common practice to run multiple environments at once instead, and wait until there's enough variety to calculate policy and value updates without too much correlation
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-22T10:08:00.577Z
Answers.LastActivityDate: 2022-02-22T10:08:00.577Z
Answers.UserDisplayName: reza karbasi
Answers.Body: <p>I ran into this and learned that not only is observation normalization is as important as reward normalization.</p>
  <p>The y-axis is showing this env reward is much higher than 1.0</p>
  <p>Try rescaling your env reward such that it always falls within the range -1 to 1.</p>
  <p>The reason why reward normalization is required is it affects the policy loss scaling. If you look at your policy loss you will also notice it much higher than 1.0. A policy loss that high makes training the neural network unstable</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T02:07:11.383Z
Answers.LastActivityDate: 2023-03-17T02:07:11.383Z
Answers.UserDisplayName: Pyrolistical
Body: <p>I am training a model using A2C with stable baselines 2. When I increased the timesteps I noticed that episode rewards seem to reset (see attached plot). I don´t understand where these sudden decays or resets could come from and I am looking for practical experience or pointers to theory what these resets could imply.</p>
  <p><a href="https://i.sstatic.net/deKCT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/deKCT.png" alt="Plot of eposide rewards" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T09:31:21.130Z
Comments.Text: could you please add some more information, e.g. what environment are you using, what is the reward space for that env, what exactly is the plot showing?
Comments.UserDisplayName: pi-tau
Tags: <reinforcement-learning><advantage-actor-critic><stable-baselines>
Title: A2C: Why do episode rewards reset?
UserDisplayName: qwertz
ThreadText: # A2C: Why do episode rewards reset?
  
  (Asked by: username_0 on 2022-02-21)
  
  I am training a model using A2C with stable baselines 2. When I increased the timesteps I noticed that episode rewards seem to reset (see attached plot). I don´t understand where these sudden decays or resets could come from and I am looking for practical experience or pointers to theory what these resets could imply.
  
  [![Plot of eposide rewards](https://i.sstatic.net/deKCT.png)](https://i.sstatic.net/deKCT.png)
  
  **Tags:** reinforcement-learning, advantage-actor-critic, stable-baselines
  > **Comments:**
  > **username_1**: could you please add some more information, e.g. what environment are you using, what is the reward space for that env, what exactly is the plot showing?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-02-22. Score: 0
  
  i've seen this in learning using deep Q network. some tips may help you came over this problem :
  
  * use some remembering mechanisms like replay buffer. some times agent forget what it has been learnt. reply buffer remembers the agent what he saw at several episodes ago.
  * something else that worked for me was changing the optimizer. As DQN article says , using RMSprop is very useful for learning neural network based agents.
  
  if these tips dont help you, please give more information about your agent.
  
  > **Comments:**
  > **username_0**: thank you. Indeed trying DQN instead of A2C seems to help. If I use a large replay buffer with DQN I don't get the resets, while with a smaller replay buffer I still get resets. Could this effect also be related to the very large variance I seem to get in episode rewards? (the individual points in the plot have a wide distribution along they y axis)  
  > **username_2**: this question needs more investigate on the network. But maybe it's right because input-output normalization of a network may affect the network. ‌Besides changing the optimizer may help you too. I suggest test RMSprop too. My intuition says me that RMSprop have too flexibility at the begging and then becomes more stable at the end of learning. this helps RL agent to not forget the first experiences.  
  > **username_3**: A2C is on-policy,  so cannot use a long term replay table. It is common practice to run multiple environments at once instead, and wait until there's enough variety to calculate policy and value updates without too much correlation  
  
  ---
  
  ### Answer by username_4 on 2023-03-17. Score: 0
  
  I ran into this and learned that not only is observation normalization is as important as reward normalization.
  
  The y-axis is showing this env reward is much higher than 1.0
  
  Try rescaling your env reward such that it always falls within the range -1 to 1.
  
  The reason why reward normalization is required is it affects the policy loss scaling. If you look at your policy loss you will also notice it much higher than 1.0. A policy loss that high makes training the neural network unstable
  
  
  ---
  
  