simon_thom:

This means that a batch size of 1 is not a problem.


A small batch size might be a problem for batch norm layers, as the estimate of the mean and std might be quite off. If you can’t fit more than a single sample into a batch you might want to try other normalization layers, e.g. InstanceNorm (or GroupNorm for small batch sizes).
The train function is re-initializing the optimizer in each iteration, which will reset the internal buffers of Adam. I would recommend passing the criterion and optimizer to this train method or just call the training procedure in the DataLoader loop.



 simon_thom:

If I train my model with model.train() on a training set and evaluate with model.eval() on the same training set, shouldn’t I be able to reproduce the exact same loss on the training set with model.eval()?


Not really, as some layers e.g. Dropout and BatchNorm will behave differently as explained before.