Answers.Body: <p>After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG:</p>
  
  <p>I added a reward component based on the actuator movement, i. e. the delta of actions from one step to the next.</p>
  
  <p>Excessive action changes are punished now and this could mitigate the tendency to oscillate. The solution is nnot really perfect, but it works for the moment.</p>
  
  <p>This finding is detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into <a href="https://github.com/opt12/Markov-Pilot/tree/master/thesis" rel="nofollow noreferrer">https://github.com/opt12/Markov-Pilot/tree/master/thesis</a> </p>
  
  <p>I'll be glad to get feedback on it. And I'll be glad to hear better solutions than adding a delta-punishment.</p>
  
  <p>Regards, Felix</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T09:05:12.910Z
Answers.LastActivityDate: 2020-06-03T09:05:12.910Z
Answers.UserDisplayName: opt12
Body: <p>I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in <span class="math-container">$[-1, 1]$</span>).</p>
  <p>To this end, I use a neural network and the DDPG algorithm, which shows promising results after about 20 minutes of training.</p>
  <p>I stripped down the presented state to the model to only the roll angle and the angular velocity, so that the neural network is not overwhelmed by state inputs.</p>
  <p>So it's a 2 input / 1 output model to perform the control task.</p>
  <p>In test runs, it looks mostly good, but sometimes, the controller starts thrashing, i.e. it outputs flittering commands, like in a very fast bangbang-controlm which causes a rapid movement of the elevator.</p>
  <p><a href="https://i.sstatic.net/hMZ9I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hMZ9I.png" alt="enter image description here" /></a></p>
  <p>Even though this behavior kind of maintains the desired target value, this behavior is absolutely undesirable. Instead, it should keep the output smooth. So far, I was not able to detect any special disturbance that starts this behavior. Yet it comes out of the blue.</p>
  <p><em>Does anybody have an idea or a hint (maybe a paper reference) on how to incorporate some element (maybe reward shaping during the training) to avoid such behavior? How to avoid rapid actuator movements in favor of smooth movements?</em></p>
  <p>I tried to include the last action in the presented state and add a punishment component in my reward, but this did not really help. So obviously, I do something wrong.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T13:07:15.210Z
Comments.Text: Try applying first order filter on the output. That should smooth out the control signal
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T14:18:36.050Z
Comments.Text: This should definitively work. It's kind of the engineering approach and I think I'll try that. However, there should be a way to prevent the agent from learning such behavior. So, good ideas are still welcome.
Comments.UserDisplayName: opt12
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-31T14:42:27.987Z
Comments.Text: Btw, are you learning through entire trajectory? Idk if you're familiar with adaptive control, but there, if you do online parameter estimation, you need persistent excitation, otherwise estimation problem becomes singular and model estimate drifts off. Similar thing might happen here where network keeps getting same learning examples of steady state and it 'forgets' how to control the system. You might try adding disturbance signal every now and then to excite dynamics so that experience replay has variety of different examples or you could stop learning through entire steady state.
Comments.UserDisplayName: Brale
Tags: <reinforcement-learning><rewards><ddpg><reward-shaping><reward-design>
Title: How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
UserDisplayName: opt12
ThreadText: # How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
  
  (Asked by: username_0 on 2020-01-31)
  
  I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$).
  
  To this end, I use a neural network and the DDPG algorithm, which shows promising results after about 20 minutes of training.
  
  I stripped down the presented state to the model to only the roll angle and the angular velocity, so that the neural network is not overwhelmed by state inputs.
  
  So it's a 2 input / 1 output model to perform the control task.
  
  In test runs, it looks mostly good, but sometimes, the controller starts thrashing, i.e. it outputs flittering commands, like in a very fast bangbang-controlm which causes a rapid movement of the elevator.
  
  [![enter image description here](https://i.sstatic.net/hMZ9I.png)](https://i.sstatic.net/hMZ9I.png)
  
  Even though this behavior kind of maintains the desired target value, this behavior is absolutely undesirable. Instead, it should keep the output smooth. So far, I was not able to detect any special disturbance that starts this behavior. Yet it comes out of the blue.
  
  *Does anybody have an idea or a hint (maybe a paper reference) on how to incorporate some element (maybe reward shaping during the training) to avoid such behavior? How to avoid rapid actuator movements in favor of smooth movements?*
  
  I tried to include the last action in the presented state and add a punishment component in my reward, but this did not really help. So obviously, I do something wrong.
  
  **Tags:** reinforcement-learning, rewards, ddpg, reward-shaping, reward-design
  
  > **Comments:**
  > **username_1**: Try applying first order filter on the output. That should smooth out the control signal  
  > **username_0**: This should definitively work. It's kind of the engineering approach and I think I'll try that. However, there should be a way to prevent the agent from learning such behavior. So, good ideas are still welcome.  
  > **username_1**: Btw, are you learning through entire trajectory? Idk if you're familiar with adaptive control, but there, if you do online parameter estimation, you need persistent excitation, otherwise estimation problem becomes singular and model estimate drifts off. Similar thing might happen here where network keeps getting same learning examples of steady state and it 'forgets' how to control the system. You might try adding disturbance signal every now and then to excite dynamics so that experience replay has variety of different examples or you could stop learning through entire steady state.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-06-03. Score: 1 | Accepted answer
  
  After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG:
  
  I added a reward component based on the actuator movement, i. e. the delta of actions from one step to the next.
  
  Excessive action changes are punished now and this could mitigate the tendency to oscillate. The solution is nnot really perfect, but it works for the moment.
  
  This finding is detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into <https://github.com/username_0/Markov-Pilot/tree/master/thesis>
  
  I'll be glad to get feedback on it. And I'll be glad to hear better solutions than adding a delta-punishment.
  
  Regards, Felix
  
  
  ---
  
  