FYI, latest pytorch’s batch norm modules have an option called track_running_stats.
https://pytorch.org/docs/master/nn.html#torch.nn.BatchNorm1d 5

If  track_running_stats  is set to  False , this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.