Body: <p>I have a doubt about how clipping affects the training of the RL agents.</p>
  <p>In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:</p>
  <pre><code>1  for i in training iterations
  2      action = clip(ddpg.prediction(state) * a + b, x, y)
  3      state, reward = environment(action)
  4      store action, state and reward
  5      if the number of experiences is larger than L:
  6          update the parameters of the agent
  </code></pre>
  <p>In this case, the actor NN that predicts the DDPG has a <span class="math-container">$\tanh$</span> activation in the output layer.</p>
  <p>My question is, could we add the clipping in the output layer of the actor (changing <span class="math-container">$\tanh(x)$</span> by <span class="math-container">$\operatorname{clip}(a\cdot \tanh(x)+b, x, y$</span>) in the training loop? Would the training work in that case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-11T13:05:07.133Z
Comments.Text: Hello. It may be a good idea to provide a link to the source where you took this pseudocode from, for people that want to have more context.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-11T14:05:09.230Z
Comments.Text: The code is based on the trained loop used by this [project](https://github.com/PettingZoo-Team/PettingZoo#API), you can see it on the link. I have written it using pseudo code because the question is related to the training of the agent rather than with the code
Comments.UserDisplayName: Leibniz
Tags: <reinforcement-learning><deep-rl><ddpg><tanh>
Title: Could we add clipping in the output layer of the actor in DDPG?
UserDisplayName: Leibniz
ThreadText: # Could we add clipping in the output layer of the actor in DDPG?
  
  (Asked by: username_0 on 2021-09-10)
  
  I have a doubt about how clipping affects the training of the RL agents.
  
  In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:
  
  ```
  1  for i in training iterations
  2      action = clip(ddpg.prediction(state) * a + b, x, y)
  3      state, reward = environment(action)
  4      store action, state and reward
  5      if the number of experiences is larger than L:
  6          update the parameters of the agent
  
  ```
  
  In this case, the actor NN that predicts the DDPG has a $\tanh$ activation in the output layer.
  
  My question is, could we add the clipping in the output layer of the actor (changing $\tanh(x)$ by $\operatorname{clip}(a\cdot \tanh(x)+b, x, y$) in the training loop? Would the training work in that case?
  
  **Tags:** reinforcement-learning, deep-rl, ddpg, tanh
  
  > **Comments:**
  > **username_1**: Hello. It may be a good idea to provide a link to the source where you took this pseudocode from, for people that want to have more context.  
  > **username_0**: The code is based on the trained loop used by this [project](https://github.com/PettingZoo-Team/PettingZoo#API), you can see it on the link. I have written it using pseudo code because the question is related to the training of the agent rather than with the code  
  
  ---
  
  ## 0 answers
  
  