Answers.Body: <p>Yes, I believe you can. Assume that you want to upper bound your difference to <span class="math-container">$k$</span>. Use the following function:</p>
  <p><span class="math-container">$$
  y_{t}^{\pi} = \frac{k}{2}*\tanh(Q_{t}^{\pi}(s,a))
  $$</span></p>
  <p>Here, <span class="math-container">$y_{t}^{\pi} \in [-\frac{k}{2}, \frac{k}{2}]$</span>. Hence the upper bound would be <span class="math-container">$k$</span>.
  <a href="https://www.desmos.com/calculator/lvefqyb3ap?lang=en" rel="nofollow noreferrer">Checkout this tanh graph</a>.</p>
  <p>A practical suggestion - Try to soften out the sharp edjes of tanh by using
  <span class="math-container">$$\frac{k}{2} * \tanh(\frac{2}{k} * Q_{t}^{\pi}(s,a))$$</span></p>
  <p>This is numerically safer. Because the difference between the <span class="math-container">$\tanh$</span> outputs reduces drastically when you start going to extreme values in both directions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T21:21:41.670Z
Answers.Text: Thanks. Or one can set an output clipping value to [0,T] (regarding binary reward) for horizon T, taking into account that the real Q-values <= T-t at time t.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T21:01:29.387Z
Answers.LastActivityDate: 2021-09-10T21:01:29.387Z
Answers.UserDisplayName: Abdus Khazi
Body: <p>Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in <span class="math-container">$R:S \rightarrow\{0,1\}$</span>. Let <span class="math-container">$Q_{t}^\pi(s,a)$</span> be the estimate from the neural network in the DQN. At timestep <span class="math-container">$t \leq T$</span>, where <span class="math-container">$T$</span> is the horizon of the RL task, is there any possible way to upper bound</p>
  <p><span class="math-container">$$ max_a Q_{t}^\pi(s,a) - min_a Q_{t}^\pi(s,a) $$</span></p>
  <p>where <span class="math-container">$\pi$</span> is the policy of DQN's Q-network (Neural Net regressor)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-07T20:15:22.053Z
Comments.Text: I deleted my answer. Could you make it clearer that $Q^{\pi}_t(s,a)$ is the estimate from the neural network in the DQN? Just because $\pi$ is from the estimator, it does not mean that $Q^{\pi}$ has to be - you could be asking about bounds from an oracle or assumed well-converged estimator. For the DQN I think there is no theoretical upper bound without stating a lot more conditions.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-07T20:23:11.453Z
Comments.Text: Can I clip the output of the NN estimator to be in [0, T] ? Then, the upper bound will be T. Am I wrong ?
Comments.UserDisplayName: ddaedalus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T06:18:30.617Z
Comments.Text: Yes you can, and that may be a reasonable bound to clip to, because it is theoretical min and max bounds of the system regardless of the policy. I cannot say whether that would be helpful in your project.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T11:19:56.993Z
Comments.Text: For what reason would you want to clip your q vals? I suggest for training to clip the gradient norm of the network instead of the q values. This will lead to much better results because it preserves the scaling of the gradient while reducing updates that are too big.
Comments.UserDisplayName: tnfru
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T13:24:29.247Z
Comments.Text: Because I want to create an upper bound of $Q^\pi$ in mathematical analysis
Comments.UserDisplayName: ddaedalus
Tags: <reinforcement-learning><deep-rl><dqn>
Title: Are the Q-values of DQN bounded at a single timestep?
UserDisplayName: ddaedalus
ThreadText: # Are the Q-values of DQN bounded at a single timestep?
  
  (Asked by: username_0 on 2021-09-07)
  
  Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q\_{t}^\pi(s,a)$ be the estimate from the neural network in the DQN. At timestep $t \leq T$, where $T$ is the horizon of the RL task, is there any possible way to upper bound
  
  $$ max\_a Q\_{t}^\pi(s,a) - min\_a Q\_{t}^\pi(s,a) $$
  
  where $\pi$ is the policy of DQN's Q-network (Neural Net regressor)?
  
  **Tags:** reinforcement-learning, deep-rl, dqn
  
  > **Comments:**
  > **username_1**: I deleted my answer. Could you make it clearer that $Q^{\pi}_t(s,a)$ is the estimate from the neural network in the DQN? Just because $\pi$ is from the estimator, it does not mean that $Q^{\pi}$ has to be - you could be asking about bounds from an oracle or assumed well-converged estimator. For the DQN I think there is no theoretical upper bound without stating a lot more conditions.  
  > **username_0**: Can I clip the output of the NN estimator to be in [0, T] ? Then, the upper bound will be T. Am I wrong ?  
  > **username_1**: Yes you can, and that may be a reasonable bound to clip to, because it is theoretical min and max bounds of the system regardless of the policy. I cannot say whether that would be helpful in your project.  
  > **username_2**: For what reason would you want to clip your q vals? I suggest for training to clip the gradient norm of the network instead of the q values. This will lead to much better results because it preserves the scaling of the gradient while reducing updates that are too big.  
  > **username_0**: Because I want to create an upper bound of $Q^\pi$ in mathematical analysis  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-10. Score: 2 | Accepted answer
  
  Yes, I believe you can. Assume that you want to upper bound your difference to $k$. Use the following function:
  
  $$
  y\_{t}^{\pi} = \frac{k}{2}\*\tanh(Q\_{t}^{\pi}(s,a))
  $$
  
  Here, $y\_{t}^{\pi} \in [-\frac{k}{2}, \frac{k}{2}]$. Hence the upper bound would be $k$.
  [Checkout this tanh graph](https://www.desmos.com/calculator/lvefqyb3ap?lang=en).
  
  A practical suggestion - Try to soften out the sharp edjes of tanh by using
  $$\frac{k}{2} \* \tanh(\frac{2}{k} \* Q\_{t}^{\pi}(s,a))$$
  
  This is numerically safer. Because the difference between the $\tanh$ outputs reduces drastically when you start going to extreme values in both directions.
  
  > **Comments:**
  > **username_0**: Thanks. Or one can set an output clipping value to [0,T] (regarding binary reward) for horizon T, taking into account that the real Q-values <= T-t at time t.  
  
  ---
  
  