Answers.Body: <p>I had the same problem where the reward kept decreasing and started to search for answers in the forum.</p>
  
  <p>I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard <strong>graph for rewards in validation time</strong>.</p>
  
  <p><a href="https://i.sstatic.net/kk11A.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kk11A.png" alt="Tensorboard graph for validation rewards"></a>. </p>
  
  <p>The fall continued until around 100k~ steps and did not change a lot for 250k~ steps. 
  After 350k~ th step, it slowly started to increase. <strong>Without knowing your number of steps trained</strong>, I would suggest training for more steps.</p>
  
  <p>Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-14T22:16:44.177Z
Answers.LastActivityDate: 2019-06-14T22:16:44.177Z
Answers.UserDisplayName: Oscar
Body: <p>I'm trying to train a PPO agent in a 3D balance ball environment. My action space is continuous.</p>
  <p>In the following graph, each dot shows the average reward from 100 episodes.</p>
  <p><a href="https://i.sstatic.net/uc8C7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uc8C7.png" alt="enter image description here" /></a></p>
  <p>Could this graph indicate that it's stuck at a local maximum?  Do I need to promote exploring by increasing the entropy, or does this look like a bug with my implementation?</p>
  <p>I am trying to maximize the average rewards.</p>
  <p>When I look at the loss, it does seem like it is minimizing the cost.  I thought that the loss function is dependent on the rewards because the advantage is calculated based on them.  The normalized advantage is then factored into the policy loss, which should say which direction for the policy to step toward.</p>
  <p>It seems like the actions are a bit noisy but the platform sometimes seems to try to keep the ball from falling off.</p>
  <p>Hyperparameters:</p>
  <ul>
  <li>Learning rate = 0.01</li>
  <li>Entropy Coefficient = 0.01</li>
  <li>Value Function Loss Coefficient = 0.5</li>
  <li>Gamma/Discount Factor = 0.995</li>
  <li>MiniBatch Size = 512</li>
  <li>Epochs = 3</li>
  <li>Clip Epsilon = 0.1</li>
  </ul>
  
Tags: <reinforcement-learning><policy-gradients><proximal-policy-optimization><continuous-action-spaces>
Title: If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?
UserDisplayName: Tony Ho
ThreadText: # If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?
  
  (Asked by: username_0 on 2019-05-09)
  
  I'm trying to train a PPO agent in a 3D balance ball environment. My action space is continuous.
  
  In the following graph, each dot shows the average reward from 100 episodes.
  
  [![enter image description here](https://i.sstatic.net/uc8C7.png)](https://i.sstatic.net/uc8C7.png)
  
  Could this graph indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy, or does this look like a bug with my implementation?
  
  I am trying to maximize the average rewards.
  
  When I look at the loss, it does seem like it is minimizing the cost. I thought that the loss function is dependent on the rewards because the advantage is calculated based on them. The normalized advantage is then factored into the policy loss, which should say which direction for the policy to step toward.
  
  It seems like the actions are a bit noisy but the platform sometimes seems to try to keep the ball from falling off.
  
  Hyperparameters:
  
  * Learning rate = 0.01
  * Entropy Coefficient = 0.01
  * Value Function Loss Coefficient = 0.5
  * Gamma/Discount Factor = 0.995
  * MiniBatch Size = 512
  * Epochs = 3
  * Clip Epsilon = 0.1
  
  **Tags:** reinforcement-learning, policy-gradients, proximal-policy-optimization, continuous-action-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-14. Score: 1
  
  I had the same problem where the reward kept decreasing and started to search for answers in the forum.
  
  I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard **graph for rewards in validation time**.
  
  [![Tensorboard graph for validation rewards](https://i.sstatic.net/kk11A.png)](https://i.sstatic.net/kk11A.png).
  
  The fall continued until around 100k~ steps and did not change a lot for 250k~ steps.
  After 350k~ th step, it slowly started to increase. **Without knowing your number of steps trained**, I would suggest training for more steps.
  
  Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.
  
  
  ---
  
  