Answers.Body: <p>Gradient descent should be performed using the sum (<a href="https://stats.stackexchange.com/q/201452/144483">or average</a>) of the losses in the minibatch.</p>
  <p>This is in fact also how I read the pseudocode in your question, though I understand it can be confusing. Note that, in the pseudocode, <span class="math-container">$j$</span> is not specified in detail. They do not, for example, have <span class="math-container">$j$</span> ranging from <span class="math-container">$0$</span> to the size of the minibatch.</p>
  <p>When they say:</p>
  <blockquote>
  <p>Sample random minibatch of transitions <span class="math-container">$\left(\phi_{j}, a_{j}, r_{j}, \phi_{j+1}\right)$</span> from <span class="math-container">$D$</span></p>
  </blockquote>
  <p>they mean multiple transitions in the minibatch (with a minibatch size of <span class="math-container">$1$</span> being a special case), and they use the index <span class="math-container">$j$</span> to collectively refer to the entire set of indices in that randomly sampled minibatch. It's not one particular number / index, <span class="math-container">$j$</span> is a set of indices. When further lines of code do something with index <span class="math-container">$j$</span>, they actually do something with <strong>all indices</strong> <span class="math-container">$j$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-31T14:37:52.087Z
Answers.Text: This is correct. The gradient gets updated over the entire minibatch. With the exception of "online learning", which like he hits on is 1 sample per batch.
Answers.UserDisplayName: hisairnessag3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-31T11:57:15.217Z
Answers.LastActivityDate: 2022-01-02T10:03:03.703Z
Answers.LastEditDate: 2022-01-02T10:03:03.703Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I've been reading Google's DeepMind Atari <a href="http://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf" rel="nofollow noreferrer" title="RL">paper</a> and I'm trying to understand how to implement experience replay.</p>
  <p><strong>Do we update the parameters <span class="math-container">$\theta$</span> of function <span class="math-container">$Q$</span> once for all the samples of the minibatch, or do we do that for each sample of the minibatch separately?</strong></p>
  <p>According to the following code from this paper, it performs the gradient descent on loss term for the <span class="math-container">$j$</span>-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.</p>
  <p><a href="https://i.sstatic.net/Ow0HM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ow0HM.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><deep-rl><dqn><experience-replay><mini-batch-gradient-descent>
Title: When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
UserDisplayName: user491626
ThreadText: # When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
  
  (Asked by: username_0 on 2018-05-30)
  
  I've been reading Google's DeepMind Atari [paper](http://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf "RL") and I'm trying to understand how to implement experience replay.
  
  **Do we update the parameters $\theta$ of function $Q$ once for all the samples of the minibatch, or do we do that for each sample of the minibatch separately?**
  
  According to the following code from this paper, it performs the gradient descent on loss term for the $j$-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.
  
  [![enter image description here](https://i.sstatic.net/Ow0HM.png)](https://i.sstatic.net/Ow0HM.png)
  
  **Tags:** reinforcement-learning, deep-rl, dqn, experience-replay, mini-batch-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-05-31. Score: 5 | Accepted answer
  
  Gradient descent should be performed using the sum ([or average](https://stats.stackexchange.com/q/201452/144483)) of the losses in the minibatch.
  
  This is in fact also how I read the pseudocode in your question, though I understand it can be confusing. Note that, in the pseudocode, $j$ is not specified in detail. They do not, for example, have $j$ ranging from $0$ to the size of the minibatch.
  
  When they say:
  
  > Sample random minibatch of transitions $\left(\phi\_{j}, a\_{j}, r\_{j}, \phi\_{j+1}\right)$ from $D$
  
  they mean multiple transitions in the minibatch (with a minibatch size of $1$ being a special case), and they use the index $j$ to collectively refer to the entire set of indices in that randomly sampled minibatch. It's not one particular number / index, $j$ is a set of indices. When further lines of code do something with index $j$, they actually do something with **all indices** $j$.
  
  > **Comments:**
  > **username_2**: This is correct. The gradient gets updated over the entire minibatch. With the exception of "online learning", which like he hits on is 1 sample per batch.  
  
  ---
  
  