Hi All,
I have what I hope to be a simple question - when mu and variance are calculated in the batchnorm layer, are the gradients propagated to the scaling?  I.e., are the mu and var in y = (x - mu) / sqrt(var + eps) simple numbers or the gradient tracked tensors?
I’m asking because I want to implement a modified version of batchnorm using the variance of the dimension before dropout is applied.  I need to know if I need to call .detach() or not on mu and var.
Thanks!
Andy