Thanks. I will try this asap. Meanwhile, I was wondering if there is a way to set the batch stats - mean and variance to 0s and 1s so that all gradients are based on these values and we altogether avoid calculating the means and variances(which might be taking up memory)?
Also, I was following your manual implementation of the batch normalization layer(pytorch_misc/batch_norm_manual.py at master · ptrblck/pytorch_misc · GitHub). Could you tell me if it supports multi GPUs?