I encounter the case that the input to some batch norm layer are the same across a batch, and this makes the running stats too small and training not stable. I wonder if it is possible to implement with functions in pytorch using a different input to estimate the running stats. The function of batch norm seems to be implemented in C (pytorch/functional.py at master · pytorch/pytorch · GitHub).