Body: <p>I have just started playing with <em>Reinforcement learning</em> and starting from the basics I'm trying to figure out how to solve <a href="https://github.com/MartinThoma/banana-gym" rel="nofollow noreferrer"><strong>Banana Gym</strong></a> with <a href="https://github.com/NervanaSystems/coach" rel="nofollow noreferrer"><strong>coach</strong></a>.</p>
  
  <p>Essentially <code>Banana-v0</code> env represents a <em>Banana shop</em> that buys a banana for \$1 on day 1 and has 3 days to sell it for anywhere between \$0 and \$2, where lower price means a higher chance to sell. Reward is the sell price less the buy price. If it doesn't sell on day 3 the banana is discarded and reward is -1 (banana buy price, no sale proceeds). That's pretty simple. </p>
  
  <p>Ideally the algorithm should learn to set a high price on day 1 and reducing it every day if it didn't sell.</p>
  
  <p>To start I took the coach-bundled <code>CartPole_ClippedPPO.py</code> and <code>CartPole_DQN.py</code> preset files and modified them to run <code>Banana-v0</code> gym. </p>
  
  <p>The trouble is that I don't see any learning progress regardless what I try, even after running like 50,000 episodes. In comparison the CartPole gym successfully trains in under 500 episodes.</p>
  
  <p>I would have expected some improvement after 50k episodes for such a simple task like Banana. </p>
  
  <p><a href="https://i.sstatic.net/yt2nk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yt2nk.png" alt="Training progress"></a></p>
  
  <p>Is it because the <em>Banana-v0</em> rewards are not predictable? I.e. whether the banana sells or not is still determined by a random number (with success chance based on the price).</p>
  
  <p>Where should I take it from here? How to identify which Coach agent algorithm I should start with and try to tune it? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-27T08:09:34.107Z
Comments.Text: Which coach algorithm did you use? If it was one that worked well for CartPole, it may need to be changed to work with a new environment.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-27T20:30:56.843Z
Comments.Text: @NeilSlater I tried **ClippedPPO** and **DQN**. If it showed at least some improvement I could take it from there and try to tune it, but as I didn't observe any noticeable improvement I doubt these two are fit for Banana. Maybe some other algorithm would do better? Not sure which one to try.
Comments.UserDisplayName: KeepLearning
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-28T08:04:46.490Z
Comments.Text: It's probably not the choice of algorithm, but hyperparameters that is working against you. If I get time I will take a deeper look and try to answer.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><learning-algorithms><gym>
Title: Unable to train Coach for Banana-v0 Gym environment
UserDisplayName: KeepLearning
ThreadText: # Unable to train Coach for Banana-v0 Gym environment
  
  (Asked by: username_0 on 2019-11-25)
  
  I have just started playing with *Reinforcement learning* and starting from the basics I'm trying to figure out how to solve [**Banana Gym**](https://github.com/MartinThoma/banana-gym) with [**coach**](https://github.com/NervanaSystems/coach).
  
  Essentially `Banana-v0` env represents a *Banana shop* that buys a banana for \$1 on day 1 and has 3 days to sell it for anywhere between \$0 and \$2, where lower price means a higher chance to sell. Reward is the sell price less the buy price. If it doesn't sell on day 3 the banana is discarded and reward is -1 (banana buy price, no sale proceeds). That's pretty simple.
  
  Ideally the algorithm should learn to set a high price on day 1 and reducing it every day if it didn't sell.
  
  To start I took the coach-bundled `CartPole_ClippedPPO.py` and `CartPole_DQN.py` preset files and modified them to run `Banana-v0` gym.
  
  The trouble is that I don't see any learning progress regardless what I try, even after running like 50,000 episodes. In comparison the CartPole gym successfully trains in under 500 episodes.
  
  I would have expected some improvement after 50k episodes for such a simple task like Banana.
  
  [![Training progress](https://i.sstatic.net/yt2nk.png)](https://i.sstatic.net/yt2nk.png)
  
  Is it because the *Banana-v0* rewards are not predictable? I.e. whether the banana sells or not is still determined by a random number (with success chance based on the price).
  
  Where should I take it from here? How to identify which Coach agent algorithm I should start with and try to tune it?
  
  **Tags:** reinforcement-learning, learning-algorithms, gym
  
  > **Comments:**
  > **username_1**: Which coach algorithm did you use? If it was one that worked well for CartPole, it may need to be changed to work with a new environment.  
  > **username_0**: @username_1 I tried **ClippedPPO** and **DQN**. If it showed at least some improvement I could take it from there and try to tune it, but as I didn't observe any noticeable improvement I doubt these two are fit for Banana. Maybe some other algorithm would do better? Not sure which one to try.  
  > **username_1**: It's probably not the choice of algorithm, but hyperparameters that is working against you. If I get time I will take a deeper look and try to answer.  
  
  ---
  
  ## 0 answers
  
  