Answers.Body: <p>Below are some tweaks that helped me accelerate the training of DDPG on a <a href="https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Learning-Environment-Examples.md#reacher" rel="nofollow noreferrer">Reacher</a>-like environment:</p>
  
  <ul>
  <li>Reducing the neural network size, compared to the original paper. Instead of: </li>
  </ul>
  
  <blockquote>
    <p>2 hidden layers with 400 and 300 units respectively </p>
  </blockquote>
  
  <p>I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this. </p>
  
  <ul>
  <li>As suggested in the paper, I added batch normalization:</li>
  </ul>
  
  <blockquote>
    <p>... to manually scale the features so they are in similar ranges across
    environments and units. We address this issue by adapting a recent technique from deep learning
    called batch normalization (Ioffe &amp; Szegedy, 2015). This technique normalizes each dimension
    across the samples in a minibatch to have unit mean and variance.</p>
  </blockquote>
  
  <p>The implementation you used does not seem to include this.</p>
  
  <ul>
  <li>Reducing the value of <span class="math-container">$\sigma$</span>, which is a parameter of the Ornstein-Uhlenbeck process used to enable exploration. Originally, it was <span class="math-container">$0.2$</span>, I used <span class="math-container">$0.05$</span>. (I can't find where this parameter is set in your code.)</li>
  </ul>
  
  <p>I am not entirely sure if this will help in your environment, but it was just to give you some ideas.</p>
  
  <p>PS: <a href="https://github.com/udacity/deep-reinforcement-learning/tree/master/ddpg-pendulum" rel="nofollow noreferrer">Here</a> is a link to the code I followed to build DDPG and <a href="https://pasteboard.co/JaaaXuf.png" rel="nofollow noreferrer">here</a> is a plot of rewards per episode.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-24T20:35:51.693Z
Answers.Text: do the actor and critic have the same architecture in your case?
Answers.UserDisplayName: Schach21
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-24T20:42:36.663Z
Answers.Text: @Schach21 , yes, similar to the paper. I just reduced the number of neurons in the first hidden layer from 400 to 128, and in the second hidden layer from 300 to 128.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-25T17:57:11.427Z
Answers.Text: what did you reward curve look like? Did learning happen and the reward curve stay up (like in the paper)? or was it rather noisy? I included batch normalization. That helped significantly. But I am not getting a smooth learning curve. Moreover, I've never seen a smooth learning curve from DDPG besides the original paper.
Answers.UserDisplayName: Schach21
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-25T22:20:47.660Z
Answers.Text: ok, perhaps I should mention that in my case, the goal was to solve the environment based on a known benchmark mean reward. The environment is solved when the mean reward remains above a threshold for 100 consecutive episodes. When this objective has been reached, I stopped the learning process. I did not test whether the reward curve stayed up or not after that (in fact, it's possible that it doesn't). If I have the time to test this specific aspect, I'll update my answer to include the resuts.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-26T13:43:03.430Z
Answers.Text: @Schach21 , I updated the answer with a plot of rewards and the code I based my implementation on. The plot of rewards is shown for 890 episodes, where each episode lasts 1000 steps at most. As it can be seen, there is a number of drops of the reward below 30, which is the target for solving this environment.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-26T16:57:29.157Z
Answers.Text: thanks! My implementation is working as well. My reward curve looks very similar to yours.
Answers.UserDisplayName: Schach21
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-24T10:29:26.550Z
Answers.LastActivityDate: 2020-05-26T13:35:12.200Z
Answers.LastEditDate: 2020-05-26T13:35:12.200Z
Answers.UserDisplayName: user5093249
Body: <p>I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. </p>
  
  <p>I am using <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="nofollow noreferrer">DDPG</a>, but it seems extremely unstable, and so far it isn't showing much learning. I've tried to </p>
  
  <ol>
  <li>adjust the learning rate, </li>
  <li>clip the gradients, </li>
  <li>change the size of the replay buffer, </li>
  <li>different neural net architectures, using SGD and Adam, </li>
  <li>change the <span class="math-container">$\tau$</span> for the soft-update. </li>
  </ol>
  
  <p>So, I'd like to know what people's experience is with this algorithm, for the environments where it was tested on the paper but also for other environments. What values of hyperparameters worked for you? Or what did you do? How cumbersome was the fine-tuning? </p>
  
  <p>I don't think my implementation is incorrect, because I pretty much replicated <a href="https://github.com/nicehiro/ddpg/blob/master/ddpg_pytorch/agent.py" rel="nofollow noreferrer">this</a>, and every other implementation I found did exactly the same. </p>
  
  <p>(Also, I am not sure this is necessarily the best website to post this kind of question, but I decided to give a shot.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-23T11:05:13.553Z
Comments.Text: Did you test it on other environments ? Test it on some OpenAI gym environments and see if it's working
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-23T14:25:20.027Z
Comments.Text: I just tested it on cart pole. It's not quite working. So, the algorithm may not be correct. I am going to double check, but regardless, I'd like to know what people's experience is when working with these algorithms on their own environments. I've implemented SAC. It wasn't very difficult to make it work, especially because my environment is not very challenging.
Comments.UserDisplayName: Schach21
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-23T14:45:02.900Z
Comments.Text: Update to my last comment. It is working, the training is just very slow and the reward somewhat noisy, but overall, i works. I used Pendulum-v0.
Comments.UserDisplayName: Schach21
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-26T21:49:24.037Z
Comments.Text: Indeed, DDPG is extremely unstable. A huge improvement related to DDPG is certainly D4PG. You should take a look a recent paper by Gabriel Barth-Maron, Matthew W. Hoffman, and others, called Distributional Policy Gradients, published in 2018 : https://paperswithcode.com/search?q_meta=&q=Distributed+Distributional+Deterministic+Policy+Gradients
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-27T03:31:12.593Z
Comments.Text: Thanks for sharing! I'll certainly take a look at their paper. Also, for anyone reading this, please feel free to share improvements upon DDPG.
Comments.UserDisplayName: Schach21
Tags: <reinforcement-learning><implementation><ddpg><hyper-parameters>
Title: What made your DDPG implementation on your environment work?
UserDisplayName: Schach21
ThreadText: # What made your DDPG implementation on your environment work?
  
  (Asked by: username_0 on 2020-05-23)
  
  I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively.
  
  I am using [DDPG](https://arxiv.org/pdf/1509.02971.pdf), but it seems extremely unstable, and so far it isn't showing much learning. I've tried to
  
  1. adjust the learning rate,
  2. clip the gradients,
  3. change the size of the replay buffer,
  4. different neural net architectures, using SGD and Adam,
  5. change the $\tau$ for the soft-update.
  
  So, I'd like to know what people's experience is with this algorithm, for the environments where it was tested on the paper but also for other environments. What values of hyperparameters worked for you? Or what did you do? How cumbersome was the fine-tuning?
  
  I don't think my implementation is incorrect, because I pretty much replicated [this](https://github.com/nicehiro/ddpg/blob/master/ddpg_pytorch/agent.py), and every other implementation I found did exactly the same.
  
  (Also, I am not sure this is necessarily the best website to post this kind of question, but I decided to give a shot.)
  
  **Tags:** reinforcement-learning, implementation, ddpg, hyper-parameters
  > **Comments:**
  > **username_1**: Did you test it on other environments ? Test it on some OpenAI gym environments and see if it's working  
  > **username_0**: I just tested it on cart pole. It's not quite working. So, the algorithm may not be correct. I am going to double check, but regardless, I'd like to know what people's experience is when working with these algorithms on their own environments. I've implemented SAC. It wasn't very difficult to make it work, especially because my environment is not very challenging.  
  > **username_0**: Update to my last comment. It is working, the training is just very slow and the reward somewhat noisy, but overall, i works. I used Pendulum-v0.  
  > **username_2**: Indeed, DDPG is extremely unstable. A huge improvement related to DDPG is certainly D4PG. You should take a look a recent paper by Gabriel Barth-Maron, Matthew W. Hoffman, and others, called Distributional Policy Gradients, published in 2018 : https://paperswithcode.com/search?q_meta=&q=Distributed+Distributional+Deterministic+Policy+Gradients  
  > **username_0**: Thanks for sharing! I'll certainly take a look at their paper. Also, for anyone reading this, please feel free to share improvements upon DDPG.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-05-24. Score: 1 | Accepted answer
  
  Below are some tweaks that helped me accelerate the training of DDPG on a [Reacher](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Learning-Environment-Examples.md#reacher)-like environment:
  
  * Reducing the neural network size, compared to the original paper. Instead of:
  
  > 2 hidden layers with 400 and 300 units respectively
  
  I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this.
  
  * As suggested in the paper, I added batch normalization:
  
  > ... to manually scale the features so they are in similar ranges across
  > environments and units. We address this issue by adapting a recent technique from deep learning
  > called batch normalization (Ioffe & Szegedy, 2015). This technique normalizes each dimension
  > across the samples in a minibatch to have unit mean and variance.
  
  The implementation you used does not seem to include this.
  
  * Reducing the value of $\sigma$, which is a parameter of the Ornstein-Uhlenbeck process used to enable exploration. Originally, it was $0.2$, I used $0.05$. (I can't find where this parameter is set in your code.)
  
  I am not entirely sure if this will help in your environment, but it was just to give you some ideas.
  
  PS: [Here](https://github.com/udacity/deep-reinforcement-learning/tree/master/ddpg-pendulum) is a link to the code I followed to build DDPG and [here](https://pasteboard.co/JaaaXuf.png) is a plot of rewards per episode.
  
  > **Comments:**
  > **username_0**: do the actor and critic have the same architecture in your case?  
  > **username_3**: @Schach21 , yes, similar to the paper. I just reduced the number of neurons in the first hidden layer from 400 to 128, and in the second hidden layer from 300 to 128.  
  > **username_0**: what did you reward curve look like? Did learning happen and the reward curve stay up (like in the paper)? or was it rather noisy? I included batch normalization. That helped significantly. But I am not getting a smooth learning curve. Moreover, I've never seen a smooth learning curve from DDPG besides the original paper.  
  > **username_3**: ok, perhaps I should mention that in my case, the goal was to solve the environment based on a known benchmark mean reward. The environment is solved when the mean reward remains above a threshold for 100 consecutive episodes. When this objective has been reached, I stopped the learning process. I did not test whether the reward curve stayed up or not after that (in fact, it's possible that it doesn't). If I have the time to test this specific aspect, I'll update my answer to include the resuts.  
  > **username_3**: @Schach21 , I updated the answer with a plot of rewards and the code I based my implementation on. The plot of rewards is shown for 890 episodes, where each episode lasts 1000 steps at most. As it can be seen, there is a number of drops of the reward below 30, which is the target for solving this environment.  
  > **username_0**: thanks! My implementation is working as well. My reward curve looks very similar to yours.  
  
  ---
  
  