Body: <p>I have PPO agent for discrete action space for <code>LunarLander-v2</code> env in gym and it works well. However, when i am trying to solve continuous version of the same env - <code>LunarLanderContinuous-v2</code> it is totally failing. I guess i made some mistakes in converting algorithm to continuous version. So, my steps of changing at algorithm are:</p>
  <ol>
  <li>Change network: return <code>mu</code> and  <code>var</code> of shape <code>n_actions</code>. I have 2 different last layers for that, for <code>mu</code> i return <code>Tanh</code> of logits and for <code>var</code> i return <code>Softplus</code> of logits.</li>
  <li>Choosing action: sampling action from normal distribution with expectation <code>mu</code> and variance <code>var</code> - <code>torch.distributions.multivariate_normal.MultivariateNormal(torch.squeeze(mu), torch.torch.diag_embed(var))</code></li>
  <li>For <code>log</code> of action probability i am using <code>dist.log_prob(actions)</code></li>
  </ol>
  <p>With this small changes my algorithm totally doesn't work. Is it right steps to convert algorithm with discrete action space to algorithm with continuous action space? I really confused, because my PPO for discrete action space work very well and with only this changes it is failing. Could you please suggest what i am doing wrong here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-28T15:16:47.297Z
Comments.Text: It sounds like you did the right changes though I will look more closely at changes I did with the same environment and algorithm. One thing that might be happening is that learning with continuous actions is just harder and you could have the wrong parameters. I think you should try changing the learning rate and other hyper parameters, increase the learning time, and maybe keeping the variance constant to make sure it’s not a problem with your code.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-30T06:42:58.183Z
Comments.Text: @S2673 Thank you for response! I found a solution but unfortunately i haven't found reason yet. So, if i have 2 distinct neural networks  for 2 actions it works really fine. But, if i have at least 1 common layer(for example first layer is the common one, and 2 next layers are different for 2 action, so i have like 5 layers in my network and 2 tails, sorry if i confused:)) all just stop working at all! It is really curios for me why it happens and may be there are some papers that checking for this problem? About problem - continuous lunar lander.
Comments.UserDisplayName: Alexander Yukhimchuk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-01T18:59:23.630Z
Comments.Text: I don’t think I understand exactly, but it sounds like the problem is with the code (especially in the neural network) instead of the parameters. It still could be possible that the parameters work only for simpler networks with one action output. Try changing the learning rate and size and amount of hidden layers and make sure the code is correct. I think I have PPO code somewhere that works with LunarLander and I could share it with you if you want.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T12:36:22.053Z
Comments.Text: @S2673 Yes, please can you share your network, thanks!
Comments.UserDisplayName: Alexander Yukhimchuk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T23:01:34.363Z
Comments.Text: I put the code in [here](https://chat.stackexchange.com/rooms/123761/ppo-code). It is written in Python with PyTorch and there are no explaining comments. You probably want to focus on the hyperparameters, and the Agent class, especially the initialization, storage, and updating. Run exactly as it is, it did pretty well on continuous `LunarLander` after 50 training loops. Holding space at any time will stop the training. `agent.run(1)` will animate the environment and show you how the agent is doing and `agent.train(50)` will train for 50 training loops.
Comments.UserDisplayName: S2673
Tags: <reinforcement-learning><policy-gradients><proximal-policy-optimization><on-policy-methods><continuous-action-spaces>
Title: PPO in continuous control not working
UserDisplayName: Alexander Yukhimchuk
ThreadText: # PPO in continuous control not working
  
  (Asked by: username_0 on 2021-04-26)
  
  I have PPO agent for discrete action space for `LunarLander-v2` env in gym and it works well. However, when i am trying to solve continuous version of the same env - `LunarLanderContinuous-v2` it is totally failing. I guess i made some mistakes in converting algorithm to continuous version. So, my steps of changing at algorithm are:
  
  1. Change network: return `mu` and `var` of shape `n_actions`. I have 2 different last layers for that, for `mu` i return `Tanh` of logits and for `var` i return `Softplus` of logits.
  2. Choosing action: sampling action from normal distribution with expectation `mu` and variance `var` - `torch.distributions.multivariate_normal.MultivariateNormal(torch.squeeze(mu), torch.torch.diag_embed(var))`
  3. For `log` of action probability i am using `dist.log_prob(actions)`
  
  With this small changes my algorithm totally doesn't work. Is it right steps to convert algorithm with discrete action space to algorithm with continuous action space? I really confused, because my PPO for discrete action space work very well and with only this changes it is failing. Could you please suggest what i am doing wrong here?
  
  **Tags:** reinforcement-learning, policy-gradients, proximal-policy-optimization, on-policy-methods, continuous-action-spaces
  > **Comments:**
  > **username_1**: It sounds like you did the right changes though I will look more closely at changes I did with the same environment and algorithm. One thing that might be happening is that learning with continuous actions is just harder and you could have the wrong parameters. I think you should try changing the learning rate and other hyper parameters, increase the learning time, and maybe keeping the variance constant to make sure it’s not a problem with your code.  
  > **username_0**: @S2673 Thank you for response! I found a solution but unfortunately i haven't found reason yet. So, if i have 2 distinct neural networks  for 2 actions it works really fine. But, if i have at least 1 common layer(for example first layer is the common one, and 2 next layers are different for 2 action, so i have like 5 layers in my network and 2 tails, sorry if i confused:)) all just stop working at all! It is really curios for me why it happens and may be there are some papers that checking for this problem? About problem - continuous lunar lander.  
  > **username_1**: I don’t think I understand exactly, but it sounds like the problem is with the code (especially in the neural network) instead of the parameters. It still could be possible that the parameters work only for simpler networks with one action output. Try changing the learning rate and size and amount of hidden layers and make sure the code is correct. I think I have PPO code somewhere that works with LunarLander and I could share it with you if you want.  
  > **username_0**: @S2673 Yes, please can you share your network, thanks!  
  > **username_1**: I put the code in [here](https://chat.stackexchange.com/rooms/123761/ppo-code). It is written in Python with PyTorch and there are no explaining comments. You probably want to focus on the hyperparameters, and the Agent class, especially the initialization, storage, and updating. Run exactly as it is, it did pretty well on continuous `LunarLander` after 50 training loops. Holding space at any time will stop the training. `agent.run(1)` will animate the environment and show you how the agent is doing and `agent.train(50)` will train for 50 training loops.  
  
  ---
  
  ## 0 answers
  
  