Body: <p>Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one mini-batch, or will it be done after all the 10 mini batches have passed through?</p>
  <p>Intuitively for me, it ought to be the first one because otherwise, the only difference between Batch-GD and mini-batch GD will be the size of the batch.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-17T13:18:49.200Z
Comments.Text: In one epoch, weights will be updated 10 times for the above case.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-17T14:37:38.793Z
Comments.Text: Thanks, such a silly thing but it was bothering a lot.
Comments.UserDisplayName: Kaustubh Sharma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-29T23:00:19.043Z
Comments.Text: Are both the the weights and the gradients updated at each iteration or at each epoch?
  Or at each minibatch iteration do you only calculate the loss and later add it to the other losses and calculate the weights?
Comments.UserDisplayName: skan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-31T02:24:35.940Z
Comments.Text: What @SwaksharDeb says is correct. There are 10 batches, so 10 updates. There are 32 data points in each batch but each batch produces a single aggregate (or mean) error on the 32 data points so a single gradient is produced making a single update to the parameter values.
Comments.UserDisplayName: Vass
Tags: <neural-networks><backpropagation><gradient-descent><weights><mini-batch-gradient-descent>
Title: In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
UserDisplayName: Kaustubh Sharma
ThreadText: # In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
  
  (Asked by: username_0 on 2021-10-17)
  
  Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one mini-batch, or will it be done after all the 10 mini batches have passed through?
  
  Intuitively for me, it ought to be the first one because otherwise, the only difference between Batch-GD and mini-batch GD will be the size of the batch.
  
  **Tags:** neural-networks, backpropagation, gradient-descent, weights, mini-batch-gradient-descent
  
  > **Comments:**
  > **username_1**: In one epoch, weights will be updated 10 times for the above case.  
  > **username_0**: Thanks, such a silly thing but it was bothering a lot.  
  > **username_3**: What @username_1 says is correct. There are 10 batches, so 10 updates. There are 32 data points in each batch but each batch produces a single aggregate (or mean) error on the 32 data points so a single gradient is produced making a single update to the parameter values.  
  > **username_2**: Are both the the weights and the gradients updated at each iteration or at each epoch? Or at each minibatch iteration do you only calculate the loss and later add it to the other losses and calculate the weights?  
  
  ---
  
  ## 0 answers
  
  