Answers.Body: <p>The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)</p>
  
  <p>generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-10T08:20:18.670Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130391/discussion-on-answer-by-mshlis-what-is-a-batch-in-batch-normalization).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T16:22:19.190Z
Answers.LastActivityDate: 2019-12-27T16:22:19.190Z
Answers.UserDisplayName: mshlis
Body: <p>I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.</p>
  
  <p>Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.</p>
  
  <p>So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps. </p>
  
  <ul>
  <li>Is each feature map a "batch"? </li>
  <li>Are the 32 feature maps the "batch"? </li>
  </ul>
  
  <p>Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,</p>
  
  <ul>
  <li>Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?</li>
  </ul>
  
  <p>Or is a "batch" something else that I've missed?</p>
  
Tags: <neural-networks><batch-normalization>
Title: What is a "batch" in batch normalization?
UserDisplayName: Alexander Soare
ThreadText: # What is a "batch" in batch normalization?
  
  (Asked by: username_0 on 2019-12-27)
  
  I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.
  
  Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.
  
  So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps.
  
  * Is each feature map a "batch"?
  * Are the 32 feature maps the "batch"?
  
  Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,
  
  * Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?
  
  Or is a "batch" something else that I've missed?
  
  **Tags:** neural-networks, batch-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-27. Score: 5 | Accepted answer
  
  The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)
  
  generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/130391/discussion-on-answer-by-username_1-what-is-a-batch-in-batch-normalization).  
  
  ---
  
  