Answers.Body: <blockquote>
  <p>How many optimisation iterations are performed on a mini-batch?</p>
  </blockquote>
  <p>Just one, as you suspected.</p>
  <blockquote>
  <p>then how does an optimisation algorithm like adam work which uses past gradient information?</p>
  </blockquote>
  <p>It uses the gradient estimates from each mini-batch as its input sequence.</p>
  <blockquote>
  <p>It seems strange since then gradients from past mini-batches are being used to minimise the loss of the current mini-batch?</p>
  </blockquote>
  <p>Each mini-batch generates a gradient which is an estimate of the true gradient of the loss function over the whole dataset. There can be a lot of randomness in this, depending on the problem and the size of the minibatch, but the expected value of the gradient for any fairly sampled minibatch is the mean gradient for the whole dataset.</p>
  <p>As there is an update between each minibatch, the expected gradient <em>will</em> change, because the new parameters will generate a new loss value from a new location in the parameter space that is being searched for optimal values.</p>
  <p>Adam is not intended to calculate a single true gradient for the whole dataset or population. If you wanted that outcome, then you could use the entire dataset for each step instead of minibatches. Instead, Adam applies some second-order effects to gradient step updates:</p>
  <ul>
  <li><p>Adam does not take a direct deepest descent step, but normalises steps in each parameter against a rolling average of gradients seen so far.</p>
  </li>
  <li><p>Adam applies a form of momentum to update steps.</p>
  </li>
  </ul>
  <p>These manipulations of gradient data appear to work well in practice when optimising parameters over complex loss function spaces with many ridges, valleys and saddle points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T16:23:47.917Z
Answers.LastActivityDate: 2021-09-27T16:23:47.917Z
Answers.UserDisplayName: Neil Slater
Body: <p>I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the parameters.</p>
  <p>My question is: how many times do we adjust the parameters per mini-batch, i.e. how many optimisation iterations are performed on a mini-batch?</p>
  <p>The fact that I can't find anything in the TensorFlow documentation about this to me implies the answer is just 1 iteration per mini-batch. If this assumption is correct, then how does an optimisation algorithm, like adam, work which uses past gradient information? It seems strange, since then gradients from past mini-batches are being used to minimise the loss of the current mini-batch?</p>
  
Tags: <neural-networks><gradient-descent><adam><mini-batch-gradient-descent>
Title: How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
UserDisplayName: user50018
ThreadText: # How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
  
  (Asked by: username_0 on 2021-09-27)
  
  I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the parameters.
  
  My question is: how many times do we adjust the parameters per mini-batch, i.e. how many optimisation iterations are performed on a mini-batch?
  
  The fact that I can't find anything in the TensorFlow documentation about this to me implies the answer is just 1 iteration per mini-batch. If this assumption is correct, then how does an optimisation algorithm, like adam, work which uses past gradient information? It seems strange, since then gradients from past mini-batches are being used to minimise the loss of the current mini-batch?
  
  **Tags:** neural-networks, gradient-descent, adam, mini-batch-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-27. Score: 0 | Accepted answer
  
  > How many optimisation iterations are performed on a mini-batch?
  
  Just one, as you suspected.
  
  > then how does an optimisation algorithm like adam work which uses past gradient information?
  
  It uses the gradient estimates from each mini-batch as its input sequence.
  
  > It seems strange since then gradients from past mini-batches are being used to minimise the loss of the current mini-batch?
  
  Each mini-batch generates a gradient which is an estimate of the true gradient of the loss function over the whole dataset. There can be a lot of randomness in this, depending on the problem and the size of the minibatch, but the expected value of the gradient for any fairly sampled minibatch is the mean gradient for the whole dataset.
  
  As there is an update between each minibatch, the expected gradient *will* change, because the new parameters will generate a new loss value from a new location in the parameter space that is being searched for optimal values.
  
  Adam is not intended to calculate a single true gradient for the whole dataset or population. If you wanted that outcome, then you could use the entire dataset for each step instead of minibatches. Instead, Adam applies some second-order effects to gradient step updates:
  
  * Adam does not take a direct deepest descent step, but normalises steps in each parameter against a rolling average of gradients seen so far.
  * Adam applies a form of momentum to update steps.
  
  These manipulations of gradient data appear to work well in practice when optimising parameters over complex loss function spaces with many ridges, valleys and saddle points.
  
  
  ---
  
  