Solved by kevinj22 in post #2 
              
              
                Looking at the code here: https://pytorch.org/docs/stable/_modules/torch/nn/modules/normalization.html 
Neither group norm nor layer norm seem to maintain running averages. The description of them suggests they might: https://pytorch.org/docs/stable/nn.html?highlight=group%20norm#torch.nn.GroupNorm 
…