mbp28:

running_mean


Hi, using the dataparallel to train the model in the multi-gpu mode, BN is conducted in device-wise manner. how running mean and running variance is estimated? also in device-wise manner? or one master and multi replica? Thanks