Answers.Body: <p>Our recent work solves this problem by using the idea of a forward-looking actor.</p>
  <p>We use a neural network to forecast the next state given the current state and current action. Then plug it into the actor training with considering the value of future states. We use our idea on TD3 and make a new algorithm TD3-FORK, which solves this problem with as few as four hours. <a href="https://github.com/honghaow/FORK/tree/master/BipedalWalkerHardcore" rel="nofollow noreferrer">https://github.com/honghaow/FORK/tree/master/BipedalWalkerHardcore</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T05:36:00.813Z
Answers.Text: Very very impressive, I'm hoping to revisit this problem soon and finally get some decent results (hopefully as good as yours!)
Answers.UserDisplayName: Chris Kempen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-07T23:52:02.103Z
Answers.LastActivityDate: 2020-10-07T23:52:02.103Z
Answers.UserDisplayName: Eric
Answers.Body: <p>I've been working with a TD3 implementation for bipedal hardcore. It solved the easy version (v2 and v3) in about 300 epochs (<a href="https://github.com/QasimWani/policy-value-methods" rel="nofollow noreferrer">https://github.com/QasimWani/policy-value-methods</a>). I've been training it for hardcore and even after about 1200 episodes, it's no where close to convergence.
  Did you end up solving, and if so, what algorithm did you end up going with?
  Cheers,
  Q.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-07T16:50:09.903Z
Answers.LastActivityDate: 2020-08-07T16:50:09.903Z
Answers.UserDisplayName: Qasim Wani
Answers.Body: <p>You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs:</p>
  
  <p><a href="https://github.com/openai/gym/pull/1789" rel="nofollow noreferrer">https://github.com/openai/gym/pull/1789</a></p>
  
  <blockquote>
    <p>Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been inconsistent and incorrect, returning the furthest hit result instead of closest.</p>
  </blockquote>
  
  <p>...</p>
  
  <blockquote>
    <p>Before fix - lidar traces through ground, and hits the side of a pit,
    giving the agent the impression of a "phantom canyon" in front of the
    pit, that only appears as it approaches the pit:
    <a href="https://i.sstatic.net/scwBD.png" rel="nofollow noreferrer">https://i.sstatic.net/scwBD.png</a></p>
  </blockquote>
  
  <p>...</p>
  
  <blockquote>
    <p>After fix - lidar is stopped by terrain, even when another object is behind it: <a href="https://i.sstatic.net/Fzg0Z.png" rel="nofollow noreferrer">https://i.sstatic.net/Fzg0Z.png</a></p>
  </blockquote>
  
  <p>...</p>
  
  <blockquote>
    <p>After triple checking the docs - I've submitted a minor tweak (returning -1 instead of 1 for an object that should be ignored) - it now seems legs are correctly ignored, and the traces are accurate in all situations!</p>
  </blockquote>
  
  <p>...</p>
  
  <blockquote>
    <p>It seems to me that solutions to BipedalWalkerHardcore-v2 have not just learned to deal with the complex environment - but advanced a step ahead, and are able to deal with the complex environment and sensory hallucinations causing them to jump at the slightest hint of a cube, and keep running even when it looks like the ground is not visible below their feet, relying more on the touch sensor than the lidar, or perhaps recognising the difference in "shape" between a real pit and a "fake pit" (A real pit has a floor)</p>
  </blockquote>
  
  <p>BipedalWalkerHardcore-v2 has been bumped to BipedalWalkerHardcore-v3 with these fixes as of Jan 31, 2020.</p>
  
  <p>You might want to try retraining your agent now! (although it is still a difficult environment to solve)</p>
  
  <p>To expand on why DDPG doesn't solve it, when although buggy, BipedalWalkerHardcore-v2 <em>is</em> solvable: The solution landscape to this problem is as full of pits as the environment itself. To learn to leap over a pit in the environment for example, the agent must perform a complex sequence of actions that is difficult to discover by random chance. Each time it fails, it learns that being close to a pit is highly likely to result in a large penalty, and in an effort to maximise it's rewards, will often remain stationary with a naive method like DDPG as the rewards for doing that are higher than trying and falling into the pit once more. In short, vanilla DDPG lacks enough exploration power to find the complex series of actions required before it converges on not going near the pit. Not to mention all of the other things it needs to learn to be successful.</p>
  
  <p>Of the very few published examples that have solved it, one used Evolutionary Strategy - a gradient free method essentially trying millions of policies and evaluating them, and a custom A3C method that was tailored to solve this particular environment. Both had high computational and sample requirements.
  I speak from experience as I have personally solved this environment with a RL exploration algorithm that generalises to other environments, solves it in 4 hours on a single cpu, and can be used with any off-policy RL algorithm, but unfortunately I'm unable to publish it because of IP with the company I work for.</p>
  
  <p>TLDR; The chance of vanilla DDPG solving it are infeasibly small.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T07:47:44.167Z
Answers.Text: Thank you so much for your answer! It's great to get some succinct and useful feedback from someone who has explored this themselves, and I'm anxious to dive in and try again, especially with the new updates. +1
Answers.UserDisplayName: Chris Kempen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T16:16:47.327Z
Answers.Text: It's pretty impressive that you have an algorithm that solves something like this in 4 hours on a cpu, that must be state of the art for exploration, too bad you can't share it.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T01:19:28.543Z
Answers.Text: @Brale Thank you, I was pretty happy with the results :) I've seen recent papers by DeepMind and Academics that are starting to look into the same concept, but DeepMind's is for pixel environments and the other was gridworlds. As far as I know, it's a novel implementation for control environments. It's essentially learned exploration instead of random noise...
Answers.UserDisplayName: parrowdice
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-04T13:18:42.833Z
Answers.LastActivityDate: 2020-02-07T01:12:11.513Z
Answers.LastEditDate: 2020-02-07T01:12:11.513Z
Answers.UserDisplayName: parrowdice
Body: <p>As the question suggests, I'm trying to see if I can solve <a href="https://gym.openai.com/envs/BipedalWalkerHardcore-v2/" rel="nofollow noreferrer">OpenAI's hardcore version of their gym's bipedal walker</a> using OpenAI's <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">DDPG algorithm</a>.</p>
  
  <p>Below is a performance graph from my latest attempt, including the hyper parameters, along with some other attempts I've made. I realise it has been solved using other custom implementations (also utilising only dense layers in Tensorflow, not convolution), but I don't seem to understand why it seems so difficult to solve using OpenAI's implementation of DDPG? Can anyone please point out where I might be going wrong? Thank you so much for any help!</p>
  
  <p>Latest attempt's performance:
  <a href="https://i.sstatic.net/CD2KP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CD2KP.png" alt="DDPG performance - Bipedal Walker Hardcore v2"></a></p>
  
  <ul>
  <li>Average score: <strong>about -75 to -80</strong></li>
  <li>Env interacts: <strong>about 8.4mil (around 2600 epochs)</strong></li>
  <li>Batch size: <strong>64</strong></li>
  <li>Replay memory: <strong>1000000</strong></li>
  <li>Network: <strong>512, 256 (relu activation on inputs, tanh on outputs)</strong></li>
  <li>All other inputs left to default</li>
  </ul>
  
  <p>Similar experiments yielded similar scores (or less), and included:</p>
  
  <ul>
  <li>Network sizes of <strong>(400,300)</strong>, <strong>(256,128)</strong>, and <strong>(128,128,128)</strong></li>
  <li>Number of epochs ranging from <strong>500</strong> all the way to <strong>100000</strong></li>
  <li>Replay memory sizes all the way up to <strong>5000000</strong></li>
  <li>Batch sizes of <strong>32</strong>, <strong>64</strong>, <strong>128</strong>, and <strong>256</strong></li>
  <li>All of the above, with both <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html" rel="nofollow noreferrer">DDPG</a> as well as <a href="https://spinningup.openai.com/en/latest/algorithms/td3.html" rel="nofollow noreferrer">TD3</a></li>
  </ul>
  
  <p>Thank you so much for any help! It would be greatly appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-06T15:53:54.367Z
Comments.Text: I found this may help a little bit.
  https://www.youtube.com/watch?v=pzzP8fA5Ipg
Comments.UserDisplayName: Eric
Tags: <reinforcement-learning><tensorflow><open-ai><gym>
Title: Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?
UserDisplayName: Chris Kempen
ThreadText: # Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?
  
  (Asked by: username_0 on 2019-08-07)
  
  As the question suggests, I'm trying to see if I can solve [OpenAI's hardcore version of their gym's bipedal walker](https://gym.openai.com/envs/BipedalWalkerHardcore-v2/) using OpenAI's [DDPG algorithm](https://spinningup.openai.com/en/latest/algorithms/ddpg.html).
  
  Below is a performance graph from my latest attempt, including the hyper parameters, along with some other attempts I've made. I realise it has been solved using other custom implementations (also utilising only dense layers in Tensorflow, not convolution), but I don't seem to understand why it seems so difficult to solve using OpenAI's implementation of DDPG? Can anyone please point out where I might be going wrong? Thank you so much for any help!
  
  Latest attempt's performance:
  [![DDPG performance - Bipedal Walker Hardcore v2](https://i.sstatic.net/CD2KP.png)](https://i.sstatic.net/CD2KP.png)
  
  * Average score: **about -75 to -80**
  * Env interacts: **about 8.4mil (around 2600 epochs)**
  * Batch size: **64**
  * Replay memory: **1000000**
  * Network: **512, 256 (relu activation on inputs, tanh on outputs)**
  * All other inputs left to default
  
  Similar experiments yielded similar scores (or less), and included:
  
  * Network sizes of **(400,300)**, **(256,128)**, and **(128,128,128)**
  * Number of epochs ranging from **500** all the way to **100000**
  * Replay memory sizes all the way up to **5000000**
  * Batch sizes of **32**, **64**, **128**, and **256**
  * All of the above, with both [DDPG](https://spinningup.openai.com/en/latest/algorithms/ddpg.html) as well as [TD3](https://spinningup.openai.com/en/latest/algorithms/td3.html)
  
  Thank you so much for any help! It would be greatly appreciated!
  
  **Tags:** reinforcement-learning, tensorflow, open-ai, gym
  
  > **Comments:**
  > **username_1**: I found this may help a little bit. https://www.youtube.com/watch?v=pzzP8fA5Ipg  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2020-02-04. Score: 5 | Accepted answer
  
  You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs:
  
  <https://github.com/openai/gym/pull/1789>
  
  > Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been inconsistent and incorrect, returning the furthest hit result instead of closest.
  
  ...
  
  > Before fix - lidar traces through ground, and hits the side of a pit,
  > giving the agent the impression of a "phantom canyon" in front of the
  > pit, that only appears as it approaches the pit:
  > <https://i.sstatic.net/scwBD.png>
  
  ...
  
  > After fix - lidar is stopped by terrain, even when another object is behind it: <https://i.sstatic.net/Fzg0Z.png>
  
  ...
  
  > After triple checking the docs - I've submitted a minor tweak (returning -1 instead of 1 for an object that should be ignored) - it now seems legs are correctly ignored, and the traces are accurate in all situations!
  
  ...
  
  > It seems to me that solutions to BipedalWalkerHardcore-v2 have not just learned to deal with the complex environment - but advanced a step ahead, and are able to deal with the complex environment and sensory hallucinations causing them to jump at the slightest hint of a cube, and keep running even when it looks like the ground is not visible below their feet, relying more on the touch sensor than the lidar, or perhaps recognising the difference in "shape" between a real pit and a "fake pit" (A real pit has a floor)
  
  BipedalWalkerHardcore-v2 has been bumped to BipedalWalkerHardcore-v3 with these fixes as of Jan 31, 2020.
  
  You might want to try retraining your agent now! (although it is still a difficult environment to solve)
  
  To expand on why DDPG doesn't solve it, when although buggy, BipedalWalkerHardcore-v2 *is* solvable: The solution landscape to this problem is as full of pits as the environment itself. To learn to leap over a pit in the environment for example, the agent must perform a complex sequence of actions that is difficult to discover by random chance. Each time it fails, it learns that being close to a pit is highly likely to result in a large penalty, and in an effort to maximise it's rewards, will often remain stationary with a naive method like DDPG as the rewards for doing that are higher than trying and falling into the pit once more. In short, vanilla DDPG lacks enough exploration power to find the complex series of actions required before it converges on not going near the pit. Not to mention all of the other things it needs to learn to be successful.
  
  Of the very few published examples that have solved it, one used Evolutionary Strategy - a gradient free method essentially trying millions of policies and evaluating them, and a custom A3C method that was tailored to solve this particular environment. Both had high computational and sample requirements.
  I speak from experience as I have personally solved this environment with a RL exploration algorithm that generalises to other environments, solves it in 4 hours on a single cpu, and can be used with any off-policy RL algorithm, but unfortunately I'm unable to publish it because of IP with the company I work for.
  
  TLDR; The chance of vanilla DDPG solving it are infeasibly small.
  
  > **Comments:**
  > **username_0**: Thank you so much for your answer! It's great to get some succinct and useful feedback from someone who has explored this themselves, and I'm anxious to dive in and try again, especially with the new updates. +1  
  > **username_4**: It's pretty impressive that you have an algorithm that solves something like this in 4 hours on a cpu, that must be state of the art for exploration, too bad you can't share it.  
  > **username_3**: @username_4 Thank you, I was pretty happy with the results :) I've seen recent papers by DeepMind and Academics that are starting to look into the same concept, but DeepMind's is for pixel environments and the other was gridworlds. As far as I know, it's a novel implementation for control environments. It's essentially learned exploration instead of random noise...  
  
  ---
  
  ### Answer by username_1 on 2020-10-07. Score: 0
  
  Our recent work solves this problem by using the idea of a forward-looking actor.
  
  We use a neural network to forecast the next state given the current state and current action. Then plug it into the actor training with considering the value of future states. We use our idea on TD3 and make a new algorithm TD3-FORK, which solves this problem with as few as four hours. <https://github.com/honghaow/FORK/tree/master/BipedalWalkerHardcore>
  
  > **Comments:**
  > **username_0**: Very very impressive, I'm hoping to revisit this problem soon and finally get some decent results (hopefully as good as yours!)  
  
  ---
  
  ### Answer by username_2 on 2020-08-07. Score: 0
  
  I've been working with a TD3 implementation for bipedal hardcore. It solved the easy version (v2 and v3) in about 300 epochs (<https://github.com/username_2/policy-value-methods>). I've been training it for hardcore and even after about 1200 episodes, it's no where close to convergence.
  Did you end up solving, and if so, what algorithm did you end up going with?
  Cheers,
  Q.
  
  
  ---
  
  