Answers.Body: <p>This is not really a bug but something undesirable, assuming you have a linear activation at the output layer, which can produce negative values.</p>
  <p>If you want to prevent the your critic model from producing negative Q values, you could use a ReLU activation at the output layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-14T12:31:37.323Z
Answers.LastActivityDate: 2022-12-14T12:31:37.323Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I'm training a simple deep q-learning algorithm with no experience buffer to solve the <code>CartPole-v5</code> environment.</p>
  <p>I want to check for overestimation, therefore I'm plotting the action-state values for each episode. After 400+ episodes, I noticed that sometimes my predicted action-state values are negative. There are no negative rewards in this environment, only a <code>+1</code> reward for each step taken.</p>
  <p>Is this normal or is it somehow a bug?</p>
  <p>Here you can see what happens on different runs: the run indicated by the brown line shows negative values for a few steps.<a href="https://i.sstatic.net/0e8sq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0e8sq.png" alt="Plot" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-14T11:05:31.643Z
Comments.Text: If you’re using a NN then this would be normal, depending on how long you’ve trained for. As with any function approximators, there will always be some error, so unless you’re using an activation that forces the output to be non-negative, it is not out of the ordinary to see infeasible values (ie negative when all rewards are non-negative). It just means your model isn’t great
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><q-learning><dqn>
Title: Negative action-state values found during deep Q-learning
UserDisplayName: Gello
ThreadText: # Negative action-state values found during deep Q-learning
  
  (Asked by: username_0 on 2022-12-14)
  
  I'm training a simple deep q-learning algorithm with no experience buffer to solve the `CartPole-v5` environment.
  
  I want to check for overestimation, therefore I'm plotting the action-state values for each episode. After 400+ episodes, I noticed that sometimes my predicted action-state values are negative. There are no negative rewards in this environment, only a `+1` reward for each step taken.
  
  Is this normal or is it somehow a bug?
  
  Here you can see what happens on different runs: the run indicated by the brown line shows negative values for a few steps.[![Plot](https://i.sstatic.net/0e8sq.png)](https://i.sstatic.net/0e8sq.png)
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn
  > **Comments:**
  > **username_1**: If you’re using a NN then this would be normal, depending on how long you’ve trained for. As with any function approximators, there will always be some error, so unless you’re using an activation that forces the output to be non-negative, it is not out of the ordinary to see infeasible values (ie negative when all rewards are non-negative). It just means your model isn’t great  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-14. Score: 2 | Accepted answer
  
  This is not really a bug but something undesirable, assuming you have a linear activation at the output layer, which can produce negative values.
  
  If you want to prevent the your critic model from producing negative Q values, you could use a ReLU activation at the output layer.
  
  
  ---
  
  