Any update on this? I started using Pytorch for my RL model and I experienced too that the behavior of BatchNormalization in Tensorflow is different from nn.BatchNorm1d of PyTorch. Have you just applied LayerNorm paying attention to call .train() and .eval() in the right places?
I have found another thread 1 where it is said that BatchNorm1d when on model.train() does not use running stats to normalize the input of the layer, which seems confusing to me. Is that mean that the layer just compute running stats and normalize input only when evaluating the model? I would say that in this case the learning would be really wrong