Body: <p>So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset.</p>
  <p>It was running slow at first but I was able to get decent accuracy after 60 epochs or so.
  However, when I added BatchNormalization layers to my two fully-connected hidden layers, it started learning at like 20% accuracy immediately, but began overfitting my data so badly that after 7 epochs my validation didn't improve from 0.01, compared to 20+ testing accuracy.</p>
  <p>Why would adding these layers which are supposed to act as a regularizer, actually cause severe overfitting instead? I'm confused.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T08:35:38.490Z
Comments.Text: How big are the batches? I had a similar symptom when I accidentally applied BN to batches of size 1. BN relies on the batches being big enough that the batch average matches the training/test/validation set average.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T13:45:10.117Z
Comments.Text: Oh like 256. I just read the ResNet paper. Do you think it's the degradation problem that He et Al addressed on their paper?
Comments.UserDisplayName: Christopher Centrella
Tags: <deep-learning><overfitting><regularization><batch-normalization>
Title: Why is BatchNormalization causing severe overfitting to my data?
UserDisplayName: Christopher Centrella
ThreadText: # Why is BatchNormalization causing severe overfitting to my data?
  
  (Asked by: username_0 on 2021-08-10)
  
  So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset.
  
  It was running slow at first but I was able to get decent accuracy after 60 epochs or so.
  However, when I added BatchNormalization layers to my two fully-connected hidden layers, it started learning at like 20% accuracy immediately, but began overfitting my data so badly that after 7 epochs my validation didn't improve from 0.01, compared to 20+ testing accuracy.
  
  Why would adding these layers which are supposed to act as a regularizer, actually cause severe overfitting instead? I'm confused.
  
  **Tags:** deep-learning, overfitting, regularization, batch-normalization
  > **Comments:**
  > **username_1**: How big are the batches? I had a similar symptom when I accidentally applied BN to batches of size 1. BN relies on the batches being big enough that the batch average matches the training/test/validation set average.  
  > **username_0**: Oh like 256. I just read the ResNet paper. Do you think it's the degradation problem that He et Al addressed on their paper?  
  
  ---
  
  ## 0 answers
  
  