<p>Yes, I believe you can. Assume that you want to upper bound your difference to <span class="math-container">$k$</span>. Use the following function:</p>
<p><span class="math-container">$$
y_{t}^{\pi} = \frac{k}{2}*\tanh(Q_{t}^{\pi}(s,a))
$$</span></p>
<p>Here, <span class="math-container">$y_{t}^{\pi} \in [-\frac{k}{2}, \frac{k}{2}]$</span>. Hence the upper bound would be <span class="math-container">$k$</span>.
<a href="https://www.desmos.com/calculator/lvefqyb3ap?lang=en" rel="nofollow noreferrer">Checkout this tanh graph</a>.</p>
<p>A practical suggestion - Try to soften out the sharp edjes of tanh by using
<span class="math-container">$$\frac{k}{2} * \tanh(\frac{2}{k} * Q_{t}^{\pi}(s,a))$$</span></p>
<p>This is numerically safer. Because the difference between the <span class="math-container">$\tanh$</span> outputs reduces drastically when you start going to extreme values in both directions.</p>

CC BY-SA 4.0
2021-09-10T21:21:41.670Z
Thanks. Or one can set an output clipping value to [0,T] (regarding binary reward) for horizon T, taking into account that the real Q-values <= T-t at time t.
ddaedalus
CC BY-SA 4.0
2021-09-10T21:01:29.387Z
2021-09-10T21:01:29.387Z
Abdus Khazi
<p>Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in <span class="math-container">$R:S \rightarrow\{0,1\}$</span>. Let <span class="math-container">$Q_{t}^\pi(s,a)$</span> be the estimate from the neural network in the DQN. At timestep <span class="math-container">$t \leq T$</span>, where <span class="math-container">$T$</span> is the horizon of the RL task, is there any possible way to upper bound</p>
<p><span class="math-container">$$ max_a Q_{t}^\pi(s,a) - min_a Q_{t}^\pi(s,a) $$</span></p>
<p>where <span class="math-container">$\pi$</span> is the policy of DQN's Q-network (Neural Net regressor)?</p>

CC BY-SA 4.0
2021-09-07T20:15:22.053Z
I deleted my answer. Could you make it clearer that $Q^{\pi}_t(s,a)$ is the estimate from the neural network in the DQN? Just because $\pi$ is from the estimator, it does not mean that $Q^{\pi}$ has to be - you could be asking about bounds from an oracle or assumed well-converged estimator. For the DQN I think there is no theoretical upper bound without stating a lot more conditions.
Neil Slater
CC BY-SA 4.0
2021-09-07T20:23:11.453Z
Can I clip the output of the NN estimator to be in [0, T] ? Then, the upper bound will be T. Am I wrong ?
ddaedalus
CC BY-SA 4.0
2021-09-08T06:18:30.617Z
Yes you can, and that may be a reasonable bound to clip to, because it is theoretical min and max bounds of the system regardless of the policy. I cannot say whether that would be helpful in your project.
Neil Slater
CC BY-SA 4.0
2021-09-10T11:19:56.993Z
For what reason would you want to clip your q vals? I suggest for training to clip the gradient norm of the network instead of the q values. This will lead to much better results because it preserves the scaling of the gradient while reducing updates that are too big.
tnfru
CC BY-SA 4.0
2021-09-10T13:24:29.247Z
Because I want to create an upper bound of $Q^\pi$ in mathematical analysis
ddaedalus
<reinforcement-learning><deep-rl><dqn>
Are the Q-values of DQN bounded at a single timestep?
ddaedalus
# Are the Q-values of DQN bounded at a single timestep?

(Asked by: username_0 on 2021-09-07)

Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q\_{t}^\pi(s,a)$ be the estimate from the neural network in the DQN. At timestep $t \leq T$, where $T$ is the horizon of the RL task, is there any possible way to upper bound

$$ max\_a Q\_{t}^\pi(s,a) - min\_a Q\_{t}^\pi(s,a) $$

where $\pi$ is the policy of DQN's Q-network (Neural Net regressor)?

**Tags:** reinforcement-learning, deep-rl, dqn

> **Comments:**
> **username_1**: I deleted my answer. Could you make it clearer that $Q^{\pi}_t(s,a)$ is the estimate from the neural network in the DQN? Just because $\pi$ is from the estimator, it does not mean that $Q^{\pi}$ has to be - you could be asking about bounds from an oracle or assumed well-converged estimator. For the DQN I think there is no theoretical upper bound without stating a lot more conditions.  
> **username_0**: Can I clip the output of the NN estimator to be in [0, T] ? Then, the upper bound will be T. Am I wrong ?  
> **username_1**: Yes you can, and that may be a reasonable bound to clip to, because it is theoretical min and max bounds of the system regardless of the policy. I cannot say whether that would be helpful in your project.  
> **username_2**: For what reason would you want to clip your q vals? I suggest for training to clip the gradient norm of the network instead of the q values. This will lead to much better results because it preserves the scaling of the gradient while reducing updates that are too big.  
> **username_0**: Because I want to create an upper bound of $Q^\pi$ in mathematical analysis  

---

## 1 answers

### Answer by username_3 on 2021-09-10. Score: 2 | Accepted answer

Yes, I believe you can. Assume that you want to upper bound your difference to $k$. Use the following function:

$$
y\_{t}^{\pi} = \frac{k}{2}\*\tanh(Q\_{t}^{\pi}(s,a))
$$

Here, $y\_{t}^{\pi} \in [-\frac{k}{2}, \frac{k}{2}]$. Hence the upper bound would be $k$.
[Checkout this tanh graph](https://www.desmos.com/calculator/lvefqyb3ap?lang=en).

A practical suggestion - Try to soften out the sharp edjes of tanh by using
$$\frac{k}{2} \* \tanh(\frac{2}{k} \* Q\_{t}^{\pi}(s,a))$$

This is numerically safer. Because the difference between the $\tanh$ outputs reduces drastically when you start going to extreme values in both directions.

> **Comments:**
> **username_0**: Thanks. Or one can set an output clipping value to [0,T] (regarding binary reward) for horizon T, taking into account that the real Q-values <= T-t at time t.  

---

