Answers.Body: <p>I found the solution, it was changing reward function and using reward scaling. A little bit change in architecture and learning rate fixed the problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T09:52:33.783Z
Answers.LastActivityDate: 2021-07-13T09:52:33.783Z
Answers.UserDisplayName: K_197
Body: <p>I am using TD3 on a custom gym environment, but the problem is that the action values stick to the end. Sticking to the end values makes reward negative, to be positive it must find action values somewhere in the mid. But, the agent doesn't learn that and keeps action values to maximum.</p>
  <p>I am using one step termination environment (environment needs actions once for each episode).</p>
  <p>How can I improve my model? I want action values to be roughly within 80% of maximum values.</p>
  <p>In DDPG, we have inverted gradients, but could something similar be applied to TD3 to make action values search within legal action space more?</p>
  <p>The score decreases as episodes increases.</p>
  <p><a href="https://i.sstatic.net/wB8p9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wB8p9.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-13T22:51:03.237Z
Comments.Text: could you please elaborate more about the solution? I am facing the same problem, and I am unsure how to solve it.
Comments.UserDisplayName: Raz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-14T12:33:41.120Z
Comments.Text: What do you mean by "the action values stick to the end"?  Can you also explain why "Sticking to the end values makes reward negative" and "to be positive it must find action values somewhere in the mid" are true?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><td3>
Title: TD3 sticking to end values
UserDisplayName: K_197
ThreadText: # TD3 sticking to end values
  
  (Asked by: username_0 on 2021-07-06)
  
  I am using TD3 on a custom gym environment, but the problem is that the action values stick to the end. Sticking to the end values makes reward negative, to be positive it must find action values somewhere in the mid. But, the agent doesn't learn that and keeps action values to maximum.
  
  I am using one step termination environment (environment needs actions once for each episode).
  
  How can I improve my model? I want action values to be roughly within 80% of maximum values.
  
  In DDPG, we have inverted gradients, but could something similar be applied to TD3 to make action values search within legal action space more?
  
  The score decreases as episodes increases.
  
  [![enter image description here](https://i.sstatic.net/wB8p9.png)](https://i.sstatic.net/wB8p9.png)
  
  **Tags:** reinforcement-learning, td3
  
  > **Comments:**
  > **username_1**: could you please elaborate more about the solution? I am facing the same problem, and I am unsure how to solve it.  
  > **username_2**: What do you mean by "the action values stick to the end"?  Can you also explain why "Sticking to the end values makes reward negative" and "to be positive it must find action values somewhere in the mid" are true?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-07-13. Score: 0 | Accepted answer
  
  I found the solution, it was changing reward function and using reward scaling. A little bit change in architecture and learning rate fixed the problem.
  
  
  ---
  
  