Hey, thanks for following-up. I appreciate it.

Have a look at the Reproducibility docs 1, if not already done.

The following is included with every run:
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)
random.seed(0)
os.environ['PYTHONHASHSEED'] = str(0)


What exactly did you change in the implementation, as I’m able to get deterministic outputs just be seeing the code with my code snippet.

You are right, I modified the implementation back to yours, and it’s reproducible. It’s very strange because the nondeterminism seems to arise from rounding errors that do not happen in the same way, i.e. on 2 different runs I get:
Epoch 1/2, training loss: 0.2351639289064084, validation loss: 0.09888318485801263
Epoch 1/2, training loss: 0.23525172622356727, validation loss: 0.10040923198367216
I’ve added the following: reshape input tensor, take the mean & var across the newly introduced dimension, and then use repeat_interleave on mean & var. Finally, do the (input - mean) / sqrt(var) calculation.