Answers.Body: <p>For TD3/DDP the Q-value is derived from the expected return based on the future rewards which also depend on the policy and the dynamics of the environment including discount factor for future rewards. As the policy improves, the Q-values for certain actions may increase, but they are also <em>constrained</em> by the environment's MDP dynamics whose Bellman optimality equation's fixed point contraction property ultimately prevents the optimized Q-values from exploding.</p>
  <p>In addition, DDPG and TD3 utilize experience replay and a target network approach to stabilize learning which help prevent the policy from diverging too quickly and maintain a balance in learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-04T07:21:08.713Z
Answers.LastActivityDate: 2024-11-04T07:21:08.713Z
Answers.UserDisplayName: cinch
Answers.Body: <p><span class="math-container">$Q(s, a)$</span> is a trained model that should correspond to the true <span class="math-container">$Q$</span>. Usually, either the action space is bounded, or such <span class="math-container">$Q$</span> has some concave-like property (the limits for each <span class="math-container">$a$</span> are <span class="math-container">$\infty$</span>).</p>
  <p>This means that you are either trying to maximize a concave function, or you are dealing with some constrained setting (bounded action space)</p>
  <p>If you assume you have the exact <span class="math-container">$Q$</span>, and it's not strongly concave, than that means that the optimal action is an unbounded one</p>
  <p>If you don't have the exact <span class="math-container">$Q$</span>, but you are learning it, you usually do one step of policy gradient (slightly change the policy) and then let the <span class="math-container">$Q$</span>-function to update sometimes (delayed policy update in TD3) so that it adapts to the new policy</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-03T14:14:21.187Z
Answers.Text: But limiting the actions will not limit the q and Not always the q is concave.
Answers.UserDisplayName: Omar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-03T14:27:12.283Z
Answers.Text: @Omar please specify if you refer to the "ideal" Q or the learnt "Q"... in the first case, if the optimal Q (eht exact one) is not concave, then the optimal action is undefined, and that's ok... if I tell you "your reward is twice what you bet", your optimal bet is infinite...
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-03T11:25:04.537Z
Answers.LastActivityDate: 2024-11-03T11:25:04.537Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Balance of exploration and exploitation: The core of reinforcement learning lies in the balance of exploration and exploitation, and using a strategy loss function with negative expected Q values in combination with exploration strategies (such as noise strategies) can ensure that the use of existing knowledge does not lead to divergence of Q values while constantly exploring new actions</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-04T07:48:25.523Z
Answers.LastActivityDate: 2024-11-04T07:48:25.523Z
Answers.UserDisplayName: user88831
Body: <p>I tried to understand why TD3/DDPG use a policy loss of −E[Q(s,π(s))], which should make the policy maximize Q-values. I expected this to push Q-values to infinity over time, as there’s no explicit constraint</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-11T14:14:41.400Z
Comments.Text: Please, edit your post (both the body and title) to format your formulas with MathJax. Thanks.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-learning><q-learning><policy-gradients>
Title: Why does TD3/DDPG use −  [  (  ,  (  ) ) ] −E[Q(s,π(s))] as the policy loss without causing Q-values to go to infinity?
UserDisplayName: Omar
ThreadText: # Why does TD3/DDPG use −  [  (  ,  (  ) ) ] −E[Q(s,π(s))] as the policy loss without causing Q-values to go to infinity?
  
  (Asked by: username_0 on 2024-11-03)
  
  I tried to understand why TD3/DDPG use a policy loss of −E[Q(s,π(s))], which should make the policy maximize Q-values. I expected this to push Q-values to infinity over time, as there’s no explicit constraint
  
  **Tags:** reinforcement-learning, deep-learning, q-learning, policy-gradients
  > **Comments:**
  > **username_1**: Please, edit your post (both the body and title) to format your formulas with MathJax. Thanks.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2024-11-04. Score: 1
  
  For TD3/DDP the Q-value is derived from the expected return based on the future rewards which also depend on the policy and the dynamics of the environment including discount factor for future rewards. As the policy improves, the Q-values for certain actions may increase, but they are also *constrained* by the environment's MDP dynamics whose Bellman optimality equation's fixed point contraction property ultimately prevents the optimized Q-values from exploding.
  
  In addition, DDPG and TD3 utilize experience replay and a target network approach to stabilize learning which help prevent the policy from diverging too quickly and maintain a balance in learning.
  
  
  ---
  
  ### Answer by username_3 on 2024-11-03. Score: 0
  
  $Q(s, a)$ is a trained model that should correspond to the true $Q$. Usually, either the action space is bounded, or such $Q$ has some concave-like property (the limits for each $a$ are $\infty$).
  
  This means that you are either trying to maximize a concave function, or you are dealing with some constrained setting (bounded action space)
  
  If you assume you have the exact $Q$, and it's not strongly concave, than that means that the optimal action is an unbounded one
  
  If you don't have the exact $Q$, but you are learning it, you usually do one step of policy gradient (slightly change the policy) and then let the $Q$-function to update sometimes (delayed policy update in TD3) so that it adapts to the new policy
  
  > **Comments:**
  > **username_0**: But limiting the actions will not limit the q and Not always the q is concave.  
  > **username_3**: @Omar please specify if you refer to the "ideal" Q or the learnt "Q"... in the first case, if the optimal Q (eht exact one) is not concave, then the optimal action is undefined, and that's ok... if I tell you "your reward is twice what you bet", your optimal bet is infinite...  
  
  ---
  
  ### Answer by username_4 on 2024-11-04. Score: 0
  
  Balance of exploration and exploitation: The core of reinforcement learning lies in the balance of exploration and exploitation, and using a strategy loss function with negative expected Q values in combination with exploration strategies (such as noise strategies) can ensure that the use of existing knowledge does not lead to divergence of Q values while constantly exploring new actions
  
  
  ---
  
  