Answers.Body: <p>DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa) </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T10:27:04.273Z
Answers.Text: What does "over-approximation on Q function" mean? Can you provide at least one source (i.e. a research paper) that supports that claim?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T15:06:50.610Z
Answers.Text: Double Q-learning (Hado van Hasselt)
Answers.UserDisplayName: daidew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T15:35:35.423Z
Answers.Text: Please, just edit your answer to provide the link to the paper.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T00:17:56.047Z
Answers.Text: You’re welcome ;)
Answers.UserDisplayName: daidew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T07:27:37.527Z
Answers.LastActivityDate: 2020-06-15T07:27:37.527Z
Answers.UserDisplayName: daidew
Body: <p>Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?</p>
  
  <p>The reason why I want to know whether Q values learnt are appropriate is because I want to apply the <a href="https://arxiv.org/abs/1103.4601" rel="nofollow noreferrer">doubly robust estimator for off-policy value evaluation</a>. Using doubly robust requires a good Q value estimate to be learnt for each state.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T13:02:12.393Z
Comments.Text: To answer the question "is it normal that Q values for the non terminal states have higher values than those of the terminal states ?", you should ask yourself what the value function represents. What is the definition of the value function? After that, you should be able to answer your question ;)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T13:34:09.777Z
Comments.Text: I believe the answer is yes the Q values for non terminal states can be higher, an example would be possible if the rewards from the start state to the terminal state is positive. However, the Q values are insanely high compared to the rewards. For the tabular learning it is easier to visualise and understand the Q values. For the continuous state space however, i am not sure how to wrap my head around that.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T13:36:21.957Z
Comments.Text: My rewards are of magnitude 10^-2 - 10^-1 range. but my Q values are of magnitude 10^1 - 10^2. Hence I believe something is off with my training..
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T13:43:07.550Z
Comments.Text: Well, maybe you should tell us about the discount factor, learning rate and how many steps and episodes, etc. That could help people to understand better if there's something wrong.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T15:02:36.560Z
Comments.Text: I will try to look more into the problem first..
Comments.UserDisplayName: calveeen
Tags: <reinforcement-learning><q-learning><dqn><off-policy-methods><value-functions>
Title: How do I know that the DQN has learnt an appropriate Q function?
UserDisplayName: calveeen
ThreadText: # How do I know that the DQN has learnt an appropriate Q function?
  
  (Asked by: username_0 on 2020-06-14)
  
  Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?
  
  The reason why I want to know whether Q values learnt are appropriate is because I want to apply the [doubly robust estimator for off-policy value evaluation](https://arxiv.org/abs/1103.4601). Using doubly robust requires a good Q value estimate to be learnt for each state.
  
  **Tags:** reinforcement-learning, q-learning, dqn, off-policy-methods, value-functions
  > **Comments:**
  > **username_1**: To answer the question "is it normal that Q values for the non terminal states have higher values than those of the terminal states ?", you should ask yourself what the value function represents. What is the definition of the value function? After that, you should be able to answer your question ;)  
  > **username_0**: I believe the answer is yes the Q values for non terminal states can be higher, an example would be possible if the rewards from the start state to the terminal state is positive. However, the Q values are insanely high compared to the rewards. For the tabular learning it is easier to visualise and understand the Q values. For the continuous state space however, i am not sure how to wrap my head around that.  
  > **username_0**: My rewards are of magnitude 10^-2 - 10^-1 range. but my Q values are of magnitude 10^1 - 10^2. Hence I believe something is off with my training..  
  > **username_1**: Well, maybe you should tell us about the discount factor, learning rate and how many steps and episodes, etc. That could help people to understand better if there's something wrong.  
  > **username_0**: I will try to look more into the problem first..  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-15. Score: 1
  
  DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)
  
  > **Comments:**
  > **username_1**: What does "over-approximation on Q function" mean? Can you provide at least one source (i.e. a research paper) that supports that claim?  
  > **username_2**: Double Q-learning (Hado van Hasselt)  
  > **username_1**: Please, just edit your answer to provide the link to the paper.  
  > **username_2**: You’re welcome ;)  
  
  ---
  
  