As expected, additionally calling torch.cuda.manual_seed or torch.cuda.manual_seed_all does not make a difference because torch.manual_seed already calls it internally. torch.rand(*size, device='cuda') is provenly operating deterministically before and after training, so I think the RNG is alright. Because the errors are only slight and gradual I suspect it is an undeterministic order of operations issue (or similar), e.g. non-determinism in what order 100 numbers are internally added, leading to different floating point errors. What could be the cause? Could it be that atomicAdd issue for example?
Running on CPU the ReLU reuse and non-reuse lead to the same results, but as far as I can tell this does not allow us to safely conclude that this is also the case when running on CUDA, as demonstrated by the difference in behaviour already exhibited by the lacking determinism. It is a good first sanity check though.
Any ideas?