Hello everyone,
I am currently training my network in mini batch size using Adam optimizer.
However, because I’ve read some posts saying if i choose the right lr and lr_schedule parameters,
sgd could perform better tahn adams, I am planning to try SGD.
From my understanding, SGD updates the parameters in each sample which has batch size of 1 since batch size greater than 1 is just a normal GD.
But, I am seeing lots of pytorch code using SGD with batch size greater than 1 (e.g. 16).
How does pytorch SGD then work and update the parameters?
Also, can I still use batch normalization when i use SGD? SGD is when batch size is 1, so surely batch normalization will either not work or perform really badly.