Answers.Body: <p>I changed the rewards to be negative and positive by substructing the mean reward. </p>
  
  <p>It seems to improve the Q function boundries.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-03T09:37:05.753Z
Answers.LastActivityDate: 2019-05-03T09:37:05.753Z
Answers.UserDisplayName: BestR
Answers.Body: <blockquote>
    <p>Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps </p>
  </blockquote>
  
  <p>If your value function keeps increasing in later steps that means that the network is still learning those Q-values, you shouldn't necessarily prevent that. Your Q-values won't increase forever even if the rewards are always positive. You basically have a regression problem here and when the value of <span class="math-container">$Q(s,a)$</span> becomes very close to the predicted value of  <span class="math-container">$r+Q(s',a)$</span> value of <span class="math-container">$Q(s,a)$</span> will stop increasing by itself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T15:20:27.520Z
Answers.Text: Yes, but than the rewards are "swollen" in the Q function std. The Q function should'nt be more than 5*max_reward.
Answers.UserDisplayName: BestR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T15:41:47.247Z
Answers.Text: sorry, I didn't quite understand, are you saying that Q-value can't be more than 5 times maximum possible reward ? What makes you think that.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T16:04:00.523Z
Answers.Text: Im saying it should not be. Otherwise, the rewards will be swollen in the q function standard deviation
Answers.UserDisplayName: BestR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T18:53:09.697Z
Answers.Text: @BestR: there is no basis in theory for your statement about this limit to the action value - its value is supposed to be the discounted sum of future rewards, and that can be any arbitrary amount times individual rewards, depending on how you have set up the environment. Please could you explain more where this constraint that you want has come from, as it may help clarify your question?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-25T07:35:59.407Z
Answers.Text: I'm still a bit confused, why exactly are you calculating std of Q-values, it's not part of the DQN algorithm or are you doing that for your own statistics ? Also, what do you mean by rewards get "swollen" in  Q-value std, why exactly are you using rewards to calculate std of Q-values. Are you maybe changing your rewards based on the value of std for Q-values ?
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T15:03:49.347Z
Answers.LastActivityDate: 2019-04-24T15:03:49.347Z
Answers.UserDisplayName: Brale
Answers.Body: <ol>
  <li><p>You can use discount factor gamma less then one.</p></li>
  <li><p>You can use finite time horizon - only for states which are no farther away then T time steps reward propagate back</p></li>
  <li><p>You can use sum of rewords averaged over time for Q</p></li>
  </ol>
  
  <p>All of those are legitimate approaches.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T15:23:42.327Z
Answers.Text: @mimirror2image 1. My discounr factor is 0.9. This increasing process happens even with discount factor 0.1.
  2. How to calculate it? The network should calculate it.
  3.Sum of rewards will still be > 0, which returns to the same procedure of increasing q function
Answers.UserDisplayName: BestR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T14:45:21.490Z
Answers.LastActivityDate: 2019-04-24T14:45:21.490Z
Answers.UserDisplayName: mirror2image
Answers.Body: <p>The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
  <a href="https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389" rel="nofollow noreferrer">https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-29T17:50:54.533Z
Answers.LastActivityDate: 2019-12-29T17:50:54.533Z
Answers.UserDisplayName: taarraas
Body: <p>Following the <strong>DQN algorithm</strong> with experience replay:</p>
  <p>Store transition <span class="math-container">$\left(\phi_{t}, a_{t}, r_{t}, \phi_{t+1}\right)$</span> in <span class="math-container">$D$</span> Sample random minibatch of transitions <span class="math-container">$\left(\phi_{j}, a_{j}, r_{j}, \phi_{j+1}\right)$</span> from <span class="math-container">$D$</span> Set</p>
  <p><span class="math-container">$$y_{j}=\left\{\begin{array}{cc}r_{j} &amp; \text { if episode terminates at j+1} \\ r_{j}+\gamma \max _{d^{\prime}} \hat{Q}\left(\phi_{j+1}, a^{\prime} ; \theta^{-}\right) &amp; \text {otherwise }\end{array}\right.$$</span></p>
  <p>Perform a gradient descent step on <span class="math-container">$\left(y_{j}-Q\left(\phi, a_{j} ; \theta\right)\right)^{2}$</span> with respect to the network parameters <span class="math-container">$\theta$</span>.</p>
  <blockquote>
  <p>We calculate the <span class="math-container">$loss=(Q(s,a)-(r+Q(s+1,a)))^2$</span>.</p>
  </blockquote>
  <blockquote>
  <p>Assume I have positive but changing rewards. Meaning, <span class="math-container">$r&gt;0$</span>.</p>
  </blockquote>
  <p>Thus, since the rewards are positive, by calculating the loss, I notice that almost always <span class="math-container">$Q(s)&lt; Q(s+1)+r$</span>.</p>
  <p>Therefore, the network learns to always increase the <span class="math-container">$Q$</span> function , and eventually, the <span class="math-container">$Q$</span> function is higher in same states in later learning steps.</p>
  <p><em>How can I stabilize the learning process?</em></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-24T18:50:45.067Z
Comments.Text: Could you clarify whether you are working on an episodic or continuous problem?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-05T10:55:34.617Z
Comments.Text: didn't u use some negative rewards?
Comments.UserDisplayName: Dan D.
Tags: <reinforcement-learning><q-learning><dqn><objective-functions><value-functions>
Title: How to stop DQN Q function from increasing during learning?
UserDisplayName: BestR
ThreadText: # How to stop DQN Q function from increasing during learning?
  
  (Asked by: username_0 on 2019-04-24)
  
  Following the **DQN algorithm** with experience replay:
  
  Store transition $\left(\phi\_{t}, a\_{t}, r\_{t}, \phi\_{t+1}\right)$ in $D$ Sample random minibatch of transitions $\left(\phi\_{j}, a\_{j}, r\_{j}, \phi\_{j+1}\right)$ from $D$ Set
  
  $$y\_{j}=\left\{\begin{array}{cc}r\_{j} & \text { if episode terminates at j+1} \\ r\_{j}+\gamma \max \_{d^{\prime}} \hat{Q}\left(\phi\_{j+1}, a^{\prime} ; \theta^{-}\right) & \text {otherwise }\end{array}\right.$$
  
  Perform a gradient descent step on $\left(y\_{j}-Q\left(\phi, a\_{j} ; \theta\right)\right)^{2}$ with respect to the network parameters $\theta$.
  
  > We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.
  
  > Assume I have positive but changing rewards. Meaning, $r>0$.
  
  Thus, since the rewards are positive, by calculating the loss, I notice that almost always $Q(s)< Q(s+1)+r$.
  
  Therefore, the network learns to always increase the $Q$ function , and eventually, the $Q$ function is higher in same states in later learning steps.
  
  *How can I stabilize the learning process?*
  
  **Tags:** reinforcement-learning, q-learning, dqn, objective-functions, value-functions
  > **Comments:**
  > **username_1**: Could you clarify whether you are working on an episodic or continuous problem?  
  > **username_2**: didn't u use some negative rewards?  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_0 on 2019-05-03. Score: 0 | Accepted answer
  
  I changed the rewards to be negative and positive by substructing the mean reward.
  
  It seems to improve the Q function boundries.
  
  
  ---
  
  ### Answer by username_3 on 2019-04-24. Score: 1
  
  > Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps
  
  If your value function keeps increasing in later steps that means that the network is still learning those Q-values, you shouldn't necessarily prevent that. Your Q-values won't increase forever even if the rewards are always positive. You basically have a regression problem here and when the value of $Q(s,a)$ becomes very close to the predicted value of $r+Q(s',a)$ value of $Q(s,a)$ will stop increasing by itself.
  
  > **Comments:**
  > **username_0**: Yes, but than the rewards are "swollen" in the Q function std. The Q function should'nt be more than 5*max_reward.  
  > **username_3**: sorry, I didn't quite understand, are you saying that Q-value can't be more than 5 times maximum possible reward ? What makes you think that.  
  > **username_0**: Im saying it should not be. Otherwise, the rewards will be swollen in the q function standard deviation  
  > **username_1**: @BestR: there is no basis in theory for your statement about this limit to the action value - its value is supposed to be the discounted sum of future rewards, and that can be any arbitrary amount times individual rewards, depending on how you have set up the environment. Please could you explain more where this constraint that you want has come from, as it may help clarify your question?  
  > **username_3**: I'm still a bit confused, why exactly are you calculating std of Q-values, it's not part of the DQN algorithm or are you doing that for your own statistics ? Also, what do you mean by rewards get "swollen" in  Q-value std, why exactly are you using rewards to calculate std of Q-values. Are you maybe changing your rewards based on the value of std for Q-values ?  
  
  ---
  
  ### Answer by username_4 on 2019-04-24. Score: 1
  
  1. You can use discount factor gamma less then one.
  2. You can use finite time horizon - only for states which are no farther away then T time steps reward propagate back
  3. You can use sum of rewords averaged over time for Q
  
  All of those are legitimate approaches.
  
  > **Comments:**
  > **username_0**: @mimirror2image 1. My discounr factor is 0.9. This increasing process happens even with discount factor 0.1. 2. How to calculate it? The network should calculate it. 3.Sum of rewards will still be > 0, which returns to the same procedure of increasing q function  
  
  ---
  
  ### Answer by username_5 on 2019-12-29. Score: 0
  
  The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
  <https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12389>
  
  
  ---
  
  