@ptrblck ,
Thank you!
Please check the following compaison. both are in train(),  input, weight, bias are all identical. running_mean and running_var are different, and the output of BN is not identical.
I have the same understanding as yours, that is why I asked you those above questions. 

图片1743×1427 227 KB