Answers.Body: <p>The way the states are used is as follows:</p>
  <p>Typically your <span class="math-container">$Q$</span>-network will state a state as input and output scores over the action space. I.e. <span class="math-container">$Q : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$</span>. So, in your replay buffer you should store <span class="math-container">$s_t, a_t, r_{t+1}, s_{t+1}, \mbox{done}$</span> (note that done just represents where the episode ended on this transition and I add for completeness.</p>
  <p>Now, when you are doing your batch updates you sample uniformly at random from this replay buffer. This means you get <span class="math-container">$B$</span> tuples of <span class="math-container">$s_t, a_t, r_{t+1}, s_{t+1}, \mbox{done}$</span>. Now, I will assume <span class="math-container">$B=1$</span> as it is easier to explain and the extension to <span class="math-container">$B &gt; 1$</span> should be easy to see.</p>
  <p>For our state-action tuple <span class="math-container">$s_t, a_t$</span> we want to shift what the network predicts for this pair to be closer to <span class="math-container">$r_{t+1} + \gamma \arg\max_a Q(s,a)$</span>. However, our neural network only takes the state as input, and outputs a vector of scores for each action. That means we want to shift the output of our network for the state <span class="math-container">$s_t$</span> towards the target I just mentioned, <em>but only for the action <span class="math-container">$a_t$</span> that we took</em>. To do this we just calculate the target, i.e. we calculate <span class="math-container">$r_{t+1} + \gamma \arg\max_a Q(s,a)$</span>, and then we do gradient ascent like we would a normal neural network where the target vector is the same as the predicted vector everywhere except the <span class="math-container">$a_t$</span>th element, which we will change to <span class="math-container">$r_{t+1} + \gamma \arg\max_a Q(s,a)$</span>. This way, our network moves closer to our Q-learning update for only the action we want, in line with how Q-learning works.</p>
  <p>It is also worth nothing that you <em>can</em> parameterise your Neural Network to be a function <span class="math-container">$Q: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span> which would make training more in line with how tabular Q-learning but is seldom used in practice as it becomes much more expensive to compute (you have to do a forward pass for each action, rather than one forward pass per state).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T15:47:49.433Z
Answers.Text: Maybe worth noting that it is *valid* to model $\hat{q}(s,a, \theta)$ directly with NN mapping $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. That leads to a different approach using minibatches over all actions in each state for max and argmax operations (inefficient) but a simpler construction of training data. I think the approach you outline is more often used in practice though
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T15:51:49.613Z
Answers.Text: @NeilSlater I had to do this recently... can get quite messy to programme (or maybe that was just the specific paper I was having to implement...)
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T15:55:36.323Z
Answers.Text: You might make a neural network for $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ if the number of valid actions in any state is low compared to full action space and/or actions have useful traits that can be generalised over. Also of course if you are working with an RL process that doesn't need max or argmax over Q (e.g. variations of Actor-Critic)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T16:07:30.073Z
Answers.Text: It was the case of the former, but the paper was weird. The state space as they defined it also depended on the action taken -- this and some other stuff they did made it annoying to put into code.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T16:11:41.823Z
Answers.Text: It's the B > 1 that I seek clarification for. If B > 1, suppose B = 5, do we do what we would do for B = 1 but 5 times, then compute the average for those 5?
Answers.UserDisplayName: THAT_AI_GUY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T16:32:23.000Z
Answers.Text: @mamauwu ah, apologies. It just becomes a typical batch update like in normal neural networks. You could either loop through your batch and do this individually or you can do it as one update. Either way, I think the common method is to accumulate the loss for each sample in the batch, take the average, and then do backprop with it.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T16:37:36.230Z
Answers.Text: Perfect! That's the answer I was looking for. Thank you so much!
Answers.UserDisplayName: THAT_AI_GUY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T19:01:54.763Z
Answers.Text: @mamauwu if you're happy with the answer, please mark it as accepted. :-)
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T15:05:57.593Z
Answers.LastActivityDate: 2020-08-12T15:53:31.813Z
Answers.LastEditDate: 2020-08-12T15:53:31.813Z
Answers.UserDisplayName: David
Body: <p>I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural network model.</p>
  <p>Suppose there are 2 valid moves in the action space (UP or DOWN)</p>
  <p>Suppose the batch size is 5, and the 5 states are this:</p>
  <p><span class="math-container">$$[s_1, s_2, s_3, s_4, s_5]$$</span></p>
  <p>We put this batch into the neural network model and output Q values. Then we put <span class="math-container">$[s_1', s_2', s_3', s_4', s_5']$</span> into a target network.</p>
  <p>What I'm confused about is this:</p>
  <p>Each state in <span class="math-container">$[s_1, s_2, s_3, s_4, s_5]$</span> is different.</p>
  <p>Are we computing Q values for UP and DOWN for ALL 5 states after they go through the neural network?</p>
  <p>For example, <span class="math-container">$$[Q_{s_1}(\text{UP}), Q_{s_1}(\text{DOWN})],
  \\ [Q_{s_2} (\text{UP}), Q_{s_2}(\text{DOWN})], \\ 
  [Q_{s_3}(\text{UP}), Q_{s_3}(\text{DOWN})], \\ 
  [Q_{s_4}(\text{UP}), Q_{s_4}(\text{DOWN})], \\ 
  [Q_{s_5}(\text{UP}), Q_{s_5}(\text{DOWN})]$$</span></p>
  
Tags: <neural-networks><machine-learning><reinforcement-learning><q-learning><dqn>
Title: When using experience replay in reinforcement learning, which state is used for training?
UserDisplayName: THAT_AI_GUY
ThreadText: # When using experience replay in reinforcement learning, which state is used for training?
  
  (Asked by: username_0 on 2020-08-12)
  
  I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural network model.
  
  Suppose there are 2 valid moves in the action space (UP or DOWN)
  
  Suppose the batch size is 5, and the 5 states are this:
  
  $$[s\_1, s\_2, s\_3, s\_4, s\_5]$$
  
  We put this batch into the neural network model and output Q values. Then we put $[s\_1', s\_2', s\_3', s\_4', s\_5']$ into a target network.
  
  What I'm confused about is this:
  
  Each state in $[s\_1, s\_2, s\_3, s\_4, s\_5]$ is different.
  
  Are we computing Q values for UP and DOWN for ALL 5 states after they go through the neural network?
  
  For example, $$[Q\_{s\_1}(\text{UP}), Q\_{s\_1}(\text{DOWN})],
  \\ [Q\_{s\_2} (\text{UP}), Q\_{s\_2}(\text{DOWN})], \\
  [Q\_{s\_3}(\text{UP}), Q\_{s\_3}(\text{DOWN})], \\
  [Q\_{s\_4}(\text{UP}), Q\_{s\_4}(\text{DOWN})], \\
  [Q\_{s\_5}(\text{UP}), Q\_{s\_5}(\text{DOWN})]$$
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, q-learning, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-12. Score: 1 | Accepted answer
  
  The way the states are used is as follows:
  
  Typically your $Q$-network will state a state as input and output scores over the action space. I.e. $Q : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$. So, in your replay buffer you should store $s\_t, a\_t, r\_{t+1}, s\_{t+1}, \mbox{done}$ (note that done just represents where the episode ended on this transition and I add for completeness.
  
  Now, when you are doing your batch updates you sample uniformly at random from this replay buffer. This means you get $B$ tuples of $s\_t, a\_t, r\_{t+1}, s\_{t+1}, \mbox{done}$. Now, I will assume $B=1$ as it is easier to explain and the extension to $B > 1$ should be easy to see.
  
  For our state-action tuple $s\_t, a\_t$ we want to shift what the network predicts for this pair to be closer to $r\_{t+1} + \gamma \arg\max\_a Q(s,a)$. However, our neural network only takes the state as input, and outputs a vector of scores for each action. That means we want to shift the output of our network for the state $s\_t$ towards the target I just mentioned, *but only for the action $a\_t$ that we took*. To do this we just calculate the target, i.e. we calculate $r\_{t+1} + \gamma \arg\max\_a Q(s,a)$, and then we do gradient ascent like we would a normal neural network where the target vector is the same as the predicted vector everywhere except the $a\_t$th element, which we will change to $r\_{t+1} + \gamma \arg\max\_a Q(s,a)$. This way, our network moves closer to our Q-learning update for only the action we want, in line with how Q-learning works.
  
  It is also worth nothing that you *can* parameterise your Neural Network to be a function $Q: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ which would make training more in line with how tabular Q-learning but is seldom used in practice as it becomes much more expensive to compute (you have to do a forward pass for each action, rather than one forward pass per state).
  
  > **Comments:**
  > **username_2**: Maybe worth noting that it is *valid* to model $\hat{q}(s,a, \theta)$ directly with NN mapping $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. That leads to a different approach using minibatches over all actions in each state for max and argmax operations (inefficient) but a simpler construction of training data. I think the approach you outline is more often used in practice though  
  > **username_1**: @NeilSlater I had to do this recently... can get quite messy to programme (or maybe that was just the specific paper I was having to implement...)  
  > **username_2**: You might make a neural network for $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ if the number of valid actions in any state is low compared to full action space and/or actions have useful traits that can be generalised over. Also of course if you are working with an RL process that doesn't need max or argmax over Q (e.g. variations of Actor-Critic)  
  > **username_1**: It was the case of the former, but the paper was weird. The state space as they defined it also depended on the action taken -- this and some other stuff they did made it annoying to put into code.  
  > **username_0**: It's the B > 1 that I seek clarification for. If B > 1, suppose B = 5, do we do what we would do for B = 1 but 5 times, then compute the average for those 5?  
  > **username_1**: @mamauwu ah, apologies. It just becomes a typical batch update like in normal neural networks. You could either loop through your batch and do this individually or you can do it as one update. Either way, I think the common method is to accumulate the loss for each sample in the batch, take the average, and then do backprop with it.  
  > **username_0**: Perfect! That's the answer I was looking for. Thank you so much!  
  > **username_1**: @mamauwu if you're happy with the answer, please mark it as accepted. :-)  
  
  ---
  
  