Hi,
I noticed that working with a “big” network- as VGG16 there is a big difference in the gradient’s magnitude that I get performing the back-propagation sample by sample and the one that I get calculating it on the whole batch in a single time.
I don’t expect this difference since I’m using the cross entropy loss function with the default reduction=‘mean’ option. Also if the option would be set on sum, since I used a batch size of 30 I would have expected a difference of magnitude of this order, while the observed factor gap in the 2 cases is ~100000.
Also taking a batch of 2 element shows a huge difference with respect to a batch of a single element.
Why do I observe such a gap in the 2 cases? Could it be related to the presence of nn.BatchNorm2d?