Answers.Body: <p>You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If <span class="math-container">$F$</span> is a frame then you would stack 4 frames <span class="math-container">$[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$</span> and the output would be 6 Q-values for frame <span class="math-container">$F_k$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T17:58:57.340Z
Answers.Text: I still don't get the idea, when I input 4 frames It outputs 4 by 6 values, so what are these values?
Answers.UserDisplayName: dato nefaridze
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T18:07:12.577Z
Answers.Text: It outputs 4 by 6 matrix because you defined the network architecture such that it outputs 4 by 6 matrix. Tensoflow, Pytorch or whatever you used cannot read your mind and know how many outputs do you want. You need to fix your network architecture
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T09:30:59.153Z
Answers.LastActivityDate: 2020-06-05T09:30:59.153Z
Answers.UserDisplayName: Brale
Body: <p>I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?</p>
  
Tags: <q-learning><deep-rl>
Title: How to predict Q-values based on stack of frames
UserDisplayName: dato nefaridze
ThreadText: # How to predict Q-values based on stack of frames
  
  (Asked by: username_0 on 2020-06-05)
  
  I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?
  
  **Tags:** q-learning, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-05. Score: 2
  
  You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F\_{k-3}, F\_{k-2}, F\_{k-1}, F\_k]$ and the output would be 6 Q-values for frame $F\_k$.
  
  > **Comments:**
  > **username_0**: I still don't get the idea, when I input 4 frames It outputs 4 by 6 values, so what are these values?  
  > **username_1**: It outputs 4 by 6 matrix because you defined the network architecture such that it outputs 4 by 6 matrix. Tensoflow, Pytorch or whatever you used cannot read your mind and know how many outputs do you want. You need to fix your network architecture  
  
  ---
  
  