Answers.Body: <p>It looks like the agent hasn't seen the better-rewarding state enough times for its weights to have changed sufficiently enough for it to keep returning to it. Go-Explore proposed to save high-rewarding states and explicitly return to them so that the actor sees them enough times.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T22:42:04.733Z
Answers.Text: Why do you think this happens that it decides to leave a better solution and explore others?
Answers.UserDisplayName: user79474
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T23:32:42.560Z
Answers.Text: @user79474 Because computers do exactly what you tell them to do. The network you are training has slow-moving features, so of course a single good episode might not be enough. Humans, on the other hand, are able to judge how well their learning is going to intentionally steer themselves in the right direction. We are inspired by how humans think, to steer the training of the network by explicitly returning to high-rewarding states.
Answers.UserDisplayName: Richie Bendall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T02:18:45.557Z
Answers.LastActivityDate: 2024-03-03T02:18:45.557Z
Answers.UserDisplayName: Richie Bendall
Body: <p>I use PPO to train my fermenter (digital twin) to maximize enzyme (product) production.</p>
  <p>action: 1 or 0 ie. add substrate at a particular time or not based on cell and enzymes present in the tank</p>
  <p>observation: enzymes in the tank and timestep</p>
  <p>reward : enzyme_slope + cell_slope   (Cell slope gives a negative reward due to negative slope as we do not want cells to go negative)</p>
  <p>While analysing the training results I saw that the agent was able to find a solution that gave a very high cumulative reward but the agent did not explore in that direction. Whereas when I look at the kind of decisions the agent was taking towards the end of the training I see the cumulative reward is much lower than the max it was able to find before.</p>
  <p><a href="https://i.sstatic.net/8SaXq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8SaXq.png" alt="" /></a></p>
  <p>The image above clearly gives me more cumulative reward and a higher enzyme activity, agent still does not explore in that direction and decides to explore other things.
  The final solutions that I am able to find are still good but the I can see that there are better solutions explored by the agent.</p>
  <p>Why is this happening?
  How does PPO decide if a particular solution is good or not and how does it decide to take a certain path?
  Should I try other algorithms as well?</p>
  
Tags: <reinforcement-learning><python><proximal-policy-optimization><stable-baselines>
Title: Why is PPO not choosing a solution that is giving a higher cumulative reward?
UserDisplayName: user79474
ThreadText: # Why is PPO not choosing a solution that is giving a higher cumulative reward?
  
  (Asked by: username_0 on 2024-03-01)
  
  I use PPO to train my fermenter (digital twin) to maximize enzyme (product) production.
  
  action: 1 or 0 ie. add substrate at a particular time or not based on cell and enzymes present in the tank
  
  observation: enzymes in the tank and timestep
  
  reward : enzyme\_slope + cell\_slope (Cell slope gives a negative reward due to negative slope as we do not want cells to go negative)
  
  While analysing the training results I saw that the agent was able to find a solution that gave a very high cumulative reward but the agent did not explore in that direction. Whereas when I look at the kind of decisions the agent was taking towards the end of the training I see the cumulative reward is much lower than the max it was able to find before.
  
  [![](https://i.sstatic.net/8SaXq.png)](https://i.sstatic.net/8SaXq.png)
  
  The image above clearly gives me more cumulative reward and a higher enzyme activity, agent still does not explore in that direction and decides to explore other things.
  The final solutions that I am able to find are still good but the I can see that there are better solutions explored by the agent.
  
  Why is this happening?
  How does PPO decide if a particular solution is good or not and how does it decide to take a certain path?
  Should I try other algorithms as well?
  
  **Tags:** reinforcement-learning, python, proximal-policy-optimization, stable-baselines
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-03. Score: 1
  
  It looks like the agent hasn't seen the better-rewarding state enough times for its weights to have changed sufficiently enough for it to keep returning to it. Go-Explore proposed to save high-rewarding states and explicitly return to them so that the actor sees them enough times.
  
  > **Comments:**
  > **username_0**: Why do you think this happens that it decides to leave a better solution and explore others?  
  > **username_1**: @username_0 Because computers do exactly what you tell them to do. The network you are training has slow-moving features, so of course a single good episode might not be enough. Humans, on the other hand, are able to judge how well their learning is going to intentionally steer themselves in the right direction. We are inspired by how humans think, to steer the training of the network by explicitly returning to high-rewarding states.  
  
  ---
  
  