AndersML:

Is there a drawback using the running stats for data normalization in production (data leak)?
(Normalization is difficult and this normalization occur deep in the network.)


It might not be straightforward to guarantee the same batch size during the usage of this model in production. This could be an issue, if the batch stats cannot be computed e.g. from a single sample (stddev would be zero). In any case, if your workflow allows for a constant batch size, you could certainly try to use this approach.