<p>RL aims at finding a policy <span class="math-container">$\pi$</span> that maximizes <span class="math-container">$G = \mathbb{E_\pi}[\sum \gamma^i r_i]$</span></p>
<p>Note how this is an expectation, so it's scale (positive) and shift invariant... indeed, given a reward function <span class="math-container">$R(s,a,s')$</span>, a second function <span class="math-container">$R(s,a,s')' = R(s,a,s')+c$</span> would lead to the same policy <span class="math-container">$\pi$</span>, as per positive scalings</p>
<p>In other words, you would get (in the optimal regime) the same policy</p>

CC BY-SA 4.0
2024-07-17T08:54:28.737Z
Thanks, but forget about the policy, it is in the Q-values estimate in which I am interested in
Jose Antonio Martin H
CC BY-SA 4.0
2024-07-17T10:31:51.457Z
@JoseAntonioMartinH they will be half
Alberto
CC BY-SA 4.0
2024-11-30T09:25:39.000Z
thanks @alberto yes, I've developed it mathematically and it converges to Q/2
Jose Antonio Martin H
CC BY-SA 4.0
2024-11-30T23:53:10.990Z
@JoseAntonioMartinH you're welcome
Alberto
CC BY-SA 4.0
2024-06-25T14:50:02.497Z
2024-06-25T14:50:02.497Z
Alberto
<p><strong>Conditions:</strong></p>
<ol>
<li>Assume you have a DRL SAC implementation.</li>
<li>You train it as usual from Replay Buffer with uniform sampling with replacement.</li>
</ol>
<p><strong>Assume you change the target of the Q-network from:</strong></p>
<p><span class="math-container">$$T = r + \gamma Q(s',*) $$</span></p>
<p><strong>To:</strong></p>
<p><span class="math-container">$$T^* = \frac{r + \gamma Q(s',*)}{2} $$</span></p>
<p><strong>Then:</strong></p>
<ul>
<li><p>To which values will now converge the Q-values? I mean, will it converge to an average reward as in average reward RL? Diverges?</p>
</li>
<li><p>I have done some tests and I have convergence to some value, but I don't know to which value it is converging to.</p>
</li>
<li><p>Is there a simple proof that the objective: <span class="math-container">$T^*$</span> does not converge to the average reward?</p>
</li>
</ul>

CC BY-SA 4.0
2024-06-21T23:18:56.857Z
"to which value it will converge"... what does that mean??
Alberto
CC BY-SA 4.0
2024-06-24T16:40:40.383Z
In the DRL replay buffer training scenario, T converges to the expected cumulative sum of rewards (r), to which value converges T*?
Jose Antonio Martin H
<reinforcement-learning>
To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$
Jose Antonio Martin H
# To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$

(Asked by: username_0 on 2024-06-21)

**Conditions:**

1. Assume you have a DRL SAC implementation.
2. You train it as usual from Replay Buffer with uniform sampling with replacement.

**Assume you change the target of the Q-network from:**

$$T = r + \gamma Q(s',\*) $$

**To:**

$$T^\* = \frac{r + \gamma Q(s',\*)}{2} $$

**Then:**

* To which values will now converge the Q-values? I mean, will it converge to an average reward as in average reward RL? Diverges?
* I have done some tests and I have convergence to some value, but I don't know to which value it is converging to.
* Is there a simple proof that the objective: $T^\*$ does not converge to the average reward?

**Tags:** reinforcement-learning

> **Comments:**
> **username_1**: "to which value it will converge"... what does that mean??  
> **username_0**: In the DRL replay buffer training scenario, T converges to the expected cumulative sum of rewards (r), to which value converges T*?  

---

## 1 answers

### Answer by username_1 on 2024-06-25. Score: 0

RL aims at finding a policy $\pi$ that maximizes $G = \mathbb{E\_\pi}[\sum \gamma^i r\_i]$

Note how this is an expectation, so it's scale (positive) and shift invariant... indeed, given a reward function $R(s,a,s')$, a second function $R(s,a,s')' = R(s,a,s')+c$ would lead to the same policy $\pi$, as per positive scalings

In other words, you would get (in the optimal regime) the same policy

> **Comments:**
> **username_0**: Thanks, but forget about the policy, it is in the Q-values estimate in which I am interested in  
> **username_1**: @JoseAntonioMartinH they will be half  
> **username_0**: thanks @alberto yes, I've developed it mathematically and it converges to Q/2  
> **username_1**: @JoseAntonioMartinH you're welcome  

---

