Answers.Body: <p>After some research on the subject, I found a solution to my problem of steady state errors in continuous control using DDPG:</p>
  
  <p>I added a reward component based on the integral of the error. This yields maximum reward if the error integral is zero and yields lower rewards down to 0 if the integral has some value in it.</p>
  
  <p>This integral component in the reward was found to be very effective, but introduces some overshoot after set-point changes. By limiting the integral to quite small quantities, this behavior could be overcome.</p>
  
  <p>All these findings are detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into <a href="https://github.com/opt12/Markov-Pilot/tree/master/thesis" rel="nofollow noreferrer">https://github.com/opt12/Markov-Pilot/tree/master/thesis</a> </p>
  
  <p>I'll be glad to get feedback on it.</p>
  
  <p>Regards, Felix</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T09:01:49.367Z
Answers.LastActivityDate: 2020-06-03T09:01:49.367Z
Answers.UserDisplayName: opt12
Answers.Body: <p>Could you figure out a workaround for this problem of the steady-state error using DDPG?</p>
  
  <p>I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.</p>
  
  <p>It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.</p>
  
  <p>All the best,</p>
  
  <p>Wilson</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-28T07:07:57.970Z
Answers.Text: Hey Wilson, yes, I could eliminate the steady state error with DDPG sucessfully by adding an integral component to the reward. I'll post a link to my master's thesis next week, where this is described in detail.
  
  You stated, that this is a DDPG-inherent problem which my not affect e. g. PPO. Do you have any references where to read about this?
Answers.UserDisplayName: opt12
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-01T14:31:18.190Z
Answers.Text: Hi Felix! That's great news! Yes, I will definitely want to read your master's thesis and understand better how you did it. Yes, a reference I read recently is this article https://arxiv.org/abs/1804.04154, it is about the attitude control for a UAV. The author compares different RL algorithms with  a conventional PID, and you can see there that the DDPG indeed presented a steady-state error and the PPO was the one that performed the best. However, in my own experiments,  the one that worked better for me so far was the improved version of DDPG, called TD3.
Answers.UserDisplayName: wilsonsamarques
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-23T19:42:33.803Z
Answers.LastActivityDate: 2020-05-23T19:42:33.803Z
Answers.UserDisplayName: wilsonsamarques
Body: <p>Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.</p>
  
  <p>In the graphs you can see what happens: </p>
  
  <p>In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time. </p>
  
  <p><a href="https://i.sstatic.net/Dr3zC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Dr3zC.png" alt="steady state error not disappearing"></a></p>
  
  <p>In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: <span class="math-container">$r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$</span>.</p>
  
  <p>This gives me a reward that lies within the interval <span class="math-container">$]0, 1]$</span> and has a value of <span class="math-container">$0.5$</span> when the deviation <span class="math-container">$dev$</span> equals the parameter <span class="math-container">$k$</span>. (That is the parameter <span class="math-container">$k$</span> kind of indicates when <em>half of the work is done</em>)</p>
  
  <p>This reward function is relatively steep for the last fraction of the deviation from <span class="math-container">$k$</span> to <span class="math-container">$0$</span>. So it would definitely be worth the effort for the agent to eliminate the residual deviation. </p>
  
  <p>However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1.
  <a href="https://i.sstatic.net/bJPFA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bJPFA.png" alt="Yellow is the actual reward in each step"></a></p>
  
  <p>Any ideas how to push the agent into some more effort to eliminate the steady state error? 
  (A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)</p>
  
  <p>The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.</p>
  
  <p>Any ideas welcome!</p>
  
  <p>Regards,
  Felix</p>
  
Tags: <deep-rl><control-problem><ddpg>
Title: Continuous control with DDPG: How to eliminate steady state error?
UserDisplayName: opt12
ThreadText: # Continuous control with DDPG: How to eliminate steady state error?
  
  (Asked by: username_0 on 2020-03-11)
  
  Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.
  
  In the graphs you can see what happens:
  
  In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time.
  
  [![steady state error not disappearing](https://i.sstatic.net/Dr3zC.png)](https://i.sstatic.net/Dr3zC.png)
  
  In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: $r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$.
  
  This gives me a reward that lies within the interval $]0, 1]$ and has a value of $0.5$ when the deviation $dev$ equals the parameter $k$. (That is the parameter $k$ kind of indicates when *half of the work is done*)
  
  This reward function is relatively steep for the last fraction of the deviation from $k$ to $0$. So it would definitely be worth the effort for the agent to eliminate the residual deviation.
  
  However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1.
  [![Yellow is the actual reward in each step](https://i.sstatic.net/bJPFA.png)](https://i.sstatic.net/bJPFA.png)
  
  Any ideas how to push the agent into some more effort to eliminate the steady state error?
  (A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)
  
  The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.
  
  Any ideas welcome!
  
  Regards,
  Felix
  
  **Tags:** deep-rl, control-problem, ddpg
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2020-06-03. Score: 0 | Accepted answer
  
  After some research on the subject, I found a solution to my problem of steady state errors in continuous control using DDPG:
  
  I added a reward component based on the integral of the error. This yields maximum reward if the error integral is zero and yields lower rewards down to 0 if the integral has some value in it.
  
  This integral component in the reward was found to be very effective, but introduces some overshoot after set-point changes. By limiting the integral to quite small quantities, this behavior could be overcome.
  
  All these findings are detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into <https://github.com/username_0/Markov-Pilot/tree/master/thesis>
  
  I'll be glad to get feedback on it.
  
  Regards, Felix
  
  
  ---
  
  ### Answer by username_1 on 2020-05-23. Score: 0
  
  Could you figure out a workaround for this problem of the steady-state error using DDPG?
  
  I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.
  
  It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.
  
  All the best,
  
  Wilson
  
  > **Comments:**
  > **username_0**: Hey Wilson, yes, I could eliminate the steady state error with DDPG sucessfully by adding an integral component to the reward. I'll post a link to my master's thesis next week, where this is described in detail.  You stated, that this is a DDPG-inherent problem which my not affect e. g. PPO. Do you have any references where to read about this?  
  > **username_1**: Hi Felix! That's great news! Yes, I will definitely want to read your master's thesis and understand better how you did it. Yes, a reference I read recently is this article https://arxiv.org/abs/1804.04154, it is about the attitude control for a UAV. The author compares different RL algorithms with  a conventional PID, and you can see there that the DDPG indeed presented a steady-state error and the PPO was the one that performed the best. However, in my own experiments,  the one that worked better for me so far was the improved version of DDPG, called TD3.  
  
  ---
  
  