Was following this post b/c I ran into same issues training an autoencoder. I don’t know if the OP has solved the problem. but I did a test last night on a AWS GPU and cuda on w/ the parameters below gave me consistent results.
torch.backends.cudnn.deterministic = True
torch.manual_seed(999)
Further I explicitly specify model.eval() after training when computing the decoders and encoders.
Alternatively when I have, below, the results were inconsistent.
torch.backends.cudnn.deterministic = True
torch.cuda.manual_seed_all(999)
As an above poster mentioned it seems as though torch.manual_seed() applies to both cuda and cpu devices for the latest version. So if you’re not getting consistent result w/ torch.cuda.manual_seed_all, try just torch.manual_seed. This may depend on the pytorch version you have installed…Hope this helps.