Body: <p>I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am <em>double Q-learning</em>.</p>
  <p>I've noticed that, given a particular state <span class="math-container">$s$</span>, the neural network will predict nearly identical <span class="math-container">$q(s, a_i)$</span> for all actions <span class="math-container">$a_1, \dots, a_n$</span>. As the neural network gets trained, this situation gets worse.</p>
  <p>I think it is predicting the mean Q value of the state. Perhaps, the small amount of data that comprises the action is being drowned out by state input?</p>
  <p>I've considered using softmax to predict the best action, but it seems like we have nearly an unlimited number of possible actions to take, and I don't want to hard-code them.</p>
  <p>EDIT, More Detail: The state is represented by all the text detected by an OCR of a GUI program + an embedding of screenshot image of the current GUI state. While in comparison the action is just 1-2 words (the text or tooltip of the GUI element we are considering clicking on), and I'm concerned that this imbalance of representation of state vs action in the input could be causing the network to accidentally (nearly) ignore the action input.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-08T17:54:07.430Z
Comments.Text: Hi. Can you please elaborate on 'I've noticed that given a particular state it will give nearly constant estimates of action Q-values across any number of actions' part and also what do you mean by the Q-values getting worse?
Comments.UserDisplayName: MonkeyDLuffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-08T18:23:46.220Z
Comments.Text: I mean given some state S, the network will predict nearly identical Q values for all possible actions. And that the problem gets worse as training progresses.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-09T07:01:30.960Z
Comments.Text: I'm facing similar issue while trying to implement linear FA with DQN. It is because my state-action representation as vectors is bad
Comments.UserDisplayName: MonkeyDLuffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-09T12:34:52.007Z
Comments.Text: @MonkeyDLuffy : how so?
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-09T17:14:06.430Z
Comments.Text: I started with very bad representation of states as vectors and play around with it. While doing so I observed the Q values are not changing for all state, action pairs. So I asked for help [here](https://ai.stackexchange.com/questions/32333/function-approximation-for-travelling-salesman-problem)
Comments.UserDisplayName: MonkeyDLuffy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T12:21:30.390Z
Comments.Text: Given the comments above, maybe you can **edit your post** clarify how your state and actions are represented and what they are in your context. You should also explain what you mean by "the state contains a lot of data" while the action doesn't. Moreover, what do you mean by "dual Q-learning"? Do you mean "Double Q-learning"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T17:14:14.017Z
Comments.Text: Ok I updated it
Comments.UserDisplayName: profPlum
Tags: <reinforcement-learning><deep-rl><q-learning>
Title: How to deal with Q-learning having low variance in predicted Q-values?
UserDisplayName: profPlum
ThreadText: # How to deal with Q-learning having low variance in predicted Q-values?
  
  (Asked by: username_0 on 2021-11-08)
  
  I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am *double Q-learning*.
  
  I've noticed that, given a particular state $s$, the neural network will predict nearly identical $q(s, a\_i)$ for all actions $a\_1, \dots, a\_n$. As the neural network gets trained, this situation gets worse.
  
  I think it is predicting the mean Q value of the state. Perhaps, the small amount of data that comprises the action is being drowned out by state input?
  
  I've considered using softmax to predict the best action, but it seems like we have nearly an unlimited number of possible actions to take, and I don't want to hard-code them.
  
  EDIT, More Detail: The state is represented by all the text detected by an OCR of a GUI program + an embedding of screenshot image of the current GUI state. While in comparison the action is just 1-2 words (the text or tooltip of the GUI element we are considering clicking on), and I'm concerned that this imbalance of representation of state vs action in the input could be causing the network to accidentally (nearly) ignore the action input.
  
  **Tags:** reinforcement-learning, deep-rl, q-learning
  
  > **Comments:**
  > **username_1**: Hi. Can you please elaborate on 'I've noticed that given a particular state it will give nearly constant estimates of action Q-values across any number of actions' part and also what do you mean by the Q-values getting worse?  
  > **username_0**: I mean given some state S, the network will predict nearly identical Q values for all possible actions. And that the problem gets worse as training progresses.  
  > **username_1**: I'm facing similar issue while trying to implement linear FA with DQN. It is because my state-action representation as vectors is bad  
  > **username_0**: @username_1 : how so?  
  > **username_1**: I started with very bad representation of states as vectors and play around with it. While doing so I observed the Q values are not changing for all state, action pairs. So I asked for help [here](https://ai.stackexchange.com/questions/32333/function-approximation-for-travelling-salesman-problem)  
  > **username_2**: Given the comments above, maybe you can **edit your post** clarify how your state and actions are represented and what they are in your context. You should also explain what you mean by "the state contains a lot of data" while the action doesn't. Moreover, what do you mean by "dual Q-learning"? Do you mean "Double Q-learning"?  
  > **username_0**: Ok I updated it  
  
  ---
  
  ## 0 answers
  
  