Question: Is the running mean and variance updated in the forward or backward pass?
Because if it’s updated in the forward pass, the algorithm becomes much simpler, even trivial. Then, we only have to iterate over the data and do a forward pass for each sample while in train mode. No need to do complicated stuff for freezing weights then.
Does anyone know this? It’s not obvious from the code since everything is hidden inside widespread C functions.