Answers.Body: <p>RL aims at finding a policy <span class="math-container">$\pi$</span> that maximizes <span class="math-container">$G = \mathbb{E_\pi}[\sum \gamma^i r_i]$</span></p>
  <p>Note how this is an expectation, so it's scale (positive) and shift invariant... indeed, given a reward function <span class="math-container">$R(s,a,s')$</span>, a second function <span class="math-container">$R(s,a,s')' = R(s,a,s')+c$</span> would lead to the same policy <span class="math-container">$\pi$</span>, as per positive scalings</p>
  <p>In other words, you would get (in the optimal regime) the same policy</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-17T08:54:28.737Z
Answers.Text: Thanks, but forget about the policy, it is in the Q-values estimate in which I am interested in
Answers.UserDisplayName: Jose Antonio Martin H
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-17T10:31:51.457Z
Answers.Text: @JoseAntonioMartinH they will be half
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T09:25:39.000Z
Answers.Text: thanks @alberto yes, I've developed it mathematically and it converges to Q/2
Answers.UserDisplayName: Jose Antonio Martin H
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T23:53:10.990Z
Answers.Text: @JoseAntonioMartinH you're welcome
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T14:50:02.497Z
Answers.LastActivityDate: 2024-06-25T14:50:02.497Z
Answers.UserDisplayName: Alberto
Body: <p><strong>Conditions:</strong></p>
  <ol>
  <li>Assume you have a DRL SAC implementation.</li>
  <li>You train it as usual from Replay Buffer with uniform sampling with replacement.</li>
  </ol>
  <p><strong>Assume you change the target of the Q-network from:</strong></p>
  <p><span class="math-container">$$T = r + \gamma Q(s',*) $$</span></p>
  <p><strong>To:</strong></p>
  <p><span class="math-container">$$T^* = \frac{r + \gamma Q(s',*)}{2} $$</span></p>
  <p><strong>Then:</strong></p>
  <ul>
  <li><p>To which values will now converge the Q-values? I mean, will it converge to an average reward as in average reward RL? Diverges?</p>
  </li>
  <li><p>I have done some tests and I have convergence to some value, but I don't know to which value it is converging to.</p>
  </li>
  <li><p>Is there a simple proof that the objective: <span class="math-container">$T^*$</span> does not converge to the average reward?</p>
  </li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-21T23:18:56.857Z
Comments.Text: "to which value it will converge"... what does that mean??
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-24T16:40:40.383Z
Comments.Text: In the DRL replay buffer training scenario, T converges to the expected cumulative sum of rewards (r), to which value converges T*?
Comments.UserDisplayName: Jose Antonio Martin H
Tags: <reinforcement-learning>
Title: To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$
UserDisplayName: Jose Antonio Martin H
ThreadText: # To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$
  
  (Asked by: username_0 on 2024-06-21)
  
  **Conditions:**
  
  1. Assume you have a DRL SAC implementation.
  2. You train it as usual from Replay Buffer with uniform sampling with replacement.
  
  **Assume you change the target of the Q-network from:**
  
  $$T = r + \gamma Q(s',\*) $$
  
  **To:**
  
  $$T^\* = \frac{r + \gamma Q(s',\*)}{2} $$
  
  **Then:**
  
  * To which values will now converge the Q-values? I mean, will it converge to an average reward as in average reward RL? Diverges?
  * I have done some tests and I have convergence to some value, but I don't know to which value it is converging to.
  * Is there a simple proof that the objective: $T^\*$ does not converge to the average reward?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: "to which value it will converge"... what does that mean??  
  > **username_0**: In the DRL replay buffer training scenario, T converges to the expected cumulative sum of rewards (r), to which value converges T*?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-25. Score: 0
  
  RL aims at finding a policy $\pi$ that maximizes $G = \mathbb{E\_\pi}[\sum \gamma^i r\_i]$
  
  Note how this is an expectation, so it's scale (positive) and shift invariant... indeed, given a reward function $R(s,a,s')$, a second function $R(s,a,s')' = R(s,a,s')+c$ would lead to the same policy $\pi$, as per positive scalings
  
  In other words, you would get (in the optimal regime) the same policy
  
  > **Comments:**
  > **username_0**: Thanks, but forget about the policy, it is in the Q-values estimate in which I am interested in  
  > **username_1**: @JoseAntonioMartinH they will be half  
  > **username_0**: thanks @alberto yes, I've developed it mathematically and it converges to Q/2  
  > **username_1**: @JoseAntonioMartinH you're welcome  
  
  ---
  
  