evermore:

the Batch Normalization layers’ parameters use a moving average( or so-called global stats), which is not stable and accuracy enough during the earliest epochs. Is it a potential bug that PyTorch forgets to use moving average for inference? or some other reasons?


Do you mean model.eval() 2?
The dropout and BatchNorm layers have different behaviors during training and inferencing period.