Answers.Body: <blockquote>
  <p>From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)</p>
  </blockquote>
  <p>You are wrong.</p>
  <p>The weights are adjusted w.r.t. the average <em>gradient</em>, and this must be calculated using <em>individual</em> loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.</p>
  <blockquote>
  <p>But is this the same in DQNs?</p>
  </blockquote>
  <p>The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using <span class="math-container">$G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}\hat{q}(S_{t+1},a', \theta)$</span> to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.</p>
  <p>In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.</p>
  <p>If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the <code>.fit</code> function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of <span class="math-container">$(\mathbf{x}_i, \mathbf{y}_i)$</span> examples directly by sampling a minibatch. In RL you get <span class="math-container">$(\mathbf{s}_i, \mathbf{a}_i, r, \mathbf{s'}_i, done)$</span> and must <em>construct</em> the <span class="math-container">$(\mathbf{x}_i, \mathbf{y}_i)$</span> minibatch from these before passing to your <code>.fit</code> function</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T10:32:13.700Z
Answers.Text: Thank you for clarifying.
  What exactly do you mean with "batch update"?
Answers.UserDisplayName: Voß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T10:39:55.587Z
Answers.Text: @bruh: I meant minibatch update, as in the process you areasking about. Will edit
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T10:46:09.773Z
Answers.Text: To get the average gradient do I calculate for every example the gradient for every wheight, then average it over the mini-batch and then adjust the weights with the averaged gradient?
Answers.UserDisplayName: Voß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T10:56:16.650Z
Answers.Text: @bruh: Yes, that is how you would do it. Most libraries with a `.fit` method or similar will do that for you
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T10:23:10.227Z
Answers.LastActivityDate: 2021-05-21T09:45:26.293Z
Answers.LastEditDate: 2021-05-21T09:45:26.293Z
Answers.UserDisplayName: Neil Slater
Body: <p>How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch, and adjust the weights w.r.t the average loss?
  (Please correct me if I'm wrong)</p>
  <p>But is this the same in DQNs? So, you sample a batch from your memory, say 64 transitions. Do I iterate through each transition and adjust the weights &quot;on the fly&quot;, or do I calculate the average loss of the batch and THEN in a big step adjust the weights w.r.t the average batch loss?</p>
  
Tags: <reinforcement-learning><deep-learning><backpropagation><objective-functions><batch-learning>
Title: What is the difference between batches in deep Q learning and supervised learning?
UserDisplayName: Voß
ThreadText: # What is the difference between batches in deep Q learning and supervised learning?
  
  (Asked by: username_0 on 2020-01-15)
  
  How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch, and adjust the weights w.r.t the average loss?
  (Please correct me if I'm wrong)
  
  But is this the same in DQNs? So, you sample a batch from your memory, say 64 transitions. Do I iterate through each transition and adjust the weights "on the fly", or do I calculate the average loss of the batch and THEN in a big step adjust the weights w.r.t the average batch loss?
  
  **Tags:** reinforcement-learning, deep-learning, backpropagation, objective-functions, batch-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-15. Score: 0 | Accepted answer
  
  > From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)
  
  You are wrong.
  
  The weights are adjusted w.r.t. the average *gradient*, and this must be calculated using *individual* loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.
  
  > But is this the same in DQNs?
  
  The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using $G\_{t:t+1} = R\_{t+1} + \gamma \text{max}\_{a'}\hat{q}(S\_{t+1},a', \theta)$ to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.
  
  In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.
  
  If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the `.fit` function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of $(\mathbf{x}\_i, \mathbf{y}\_i)$ examples directly by sampling a minibatch. In RL you get $(\mathbf{s}\_i, \mathbf{a}\_i, r, \mathbf{s'}\_i, done)$ and must *construct* the $(\mathbf{x}\_i, \mathbf{y}\_i)$ minibatch from these before passing to your `.fit` function
  
  > **Comments:**
  > **username_0**: Thank you for clarifying. What exactly do you mean with "batch update"?  
  > **username_1**: @bruh: I meant minibatch update, as in the process you areasking about. Will edit  
  > **username_0**: To get the average gradient do I calculate for every example the gradient for every wheight, then average it over the mini-batch and then adjust the weights with the averaged gradient?  
  > **username_1**: @bruh: Yes, that is how you would do it. Most libraries with a `.fit` method or similar will do that for you  
  
  ---
  
  