Ok, it makes sense, I replaced all “p.data =” with “p.copy_” and added no_grad() context. No difference in performance that I can see, but if it’s safer, so be it.
I ran a few experiments with scaling, and yes, it seems like scaling is necessary, otherwise batchnorm will screw things up during inference. Recomputing batch statistics during inference also fixes the issue (same good accuracy with or without scaling), but that’s obviously not a solution. Not sure how to use torch.bernoulli did you mean binomial? Tried generating binomial masks, but I don’t see a good way to generate them quickly on GPU. I could only do mask = binomial_distr.sample(p.size).cuda() and this is very slow.