I trained a ResNet-20 model on cifar-10 dataset with PyTorch and Caffe respectively, and got top1-accuracy around 91.5% for both. However, quite different phenomena is observed, especially during the earliest epochs. On caffe，the earliest epochs are featured with relatively larger testing loss(e.g, several times larger than the training loss). On PyTorch, testing loss and training loss are close all the time. To my knowledge, the phenomena on caffe is rational, because during inference, the Batch Normalization layers’ parameters use a moving average( or so-called global stats), which is not stable and accuracy enough during the earliest epochs. Is it a potential bug that PyTorch forgets to use moving average for inference? or some other reasons?