<p>So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset.</p>
<p>It was running slow at first but I was able to get decent accuracy after 60 epochs or so.
However, when I added BatchNormalization layers to my two fully-connected hidden layers, it started learning at like 20% accuracy immediately, but began overfitting my data so badly that after 7 epochs my validation didn't improve from 0.01, compared to 20+ testing accuracy.</p>
<p>Why would adding these layers which are supposed to act as a regularizer, actually cause severe overfitting instead? I'm confused.</p>

CC BY-SA 4.0
2021-08-11T08:35:38.490Z
How big are the batches? I had a similar symptom when I accidentally applied BN to batches of size 1. BN relies on the batches being big enough that the batch average matches the training/test/validation set average.
Criticize SE actions means ban
CC BY-SA 4.0
2021-08-12T13:45:10.117Z
Oh like 256. I just read the ResNet paper. Do you think it's the degradation problem that He et Al addressed on their paper?
Christopher Centrella
<deep-learning><overfitting><regularization><batch-normalization>
Why is BatchNormalization causing severe overfitting to my data?
Christopher Centrella
# Why is BatchNormalization causing severe overfitting to my data?

(Asked by: username_0 on 2021-08-10)

So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset.

It was running slow at first but I was able to get decent accuracy after 60 epochs or so.
However, when I added BatchNormalization layers to my two fully-connected hidden layers, it started learning at like 20% accuracy immediately, but began overfitting my data so badly that after 7 epochs my validation didn't improve from 0.01, compared to 20+ testing accuracy.

Why would adding these layers which are supposed to act as a regularizer, actually cause severe overfitting instead? I'm confused.

**Tags:** deep-learning, overfitting, regularization, batch-normalization

> **Comments:**
> **username_1**: How big are the batches? I had a similar symptom when I accidentally applied BN to batches of size 1. BN relies on the batches being big enough that the batch average matches the training/test/validation set average.  
> **username_0**: Oh like 256. I just read the ResNet paper. Do you think it's the degradation problem that He et Al addressed on their paper?  

---

## 0 answers

