Solved by ptrblck in post #6 
              
              
                I wasn’t sure, but based on this small code snippet, it seems the latter approach is used: 

# Manual without detach
torch.manual_seed(2809)
x = torch.randn(10, 3, 4, 4, requires_grad=True)
mean = x.mean(dim=[0, 2, 3], keepdim=True)
invstd = torch.sqrt(x.var([0, 2, 3], unbiased=False, keepdim=True) …