Thank you for the reference! Yes this is super helpful. It worked for me to add these two lines:
torch.backends.cudnn.benchmark = False
torch.set_deterministic(True)

So just to summarize, there are several sources of randomness when using Pytorch:


Application-side randomness: this happens at our application level and includes: parameter initialization, data loader sampling, augmentation, and any other sampling wrote by ourselves in our own algorithms. Most often we are controlling this randomness.

np.random.seed(seed)
torch.manual_seed(seed)
random.seed(seed)

take care of this aspect.
Note: To go one step further, in DDP the DistributedSampler (which is used to split the batch across processes) uses a separate generator with its own seed to do random sampling at each epoch. It is written in a way that we (developers) usually don’t have to touch the seed. So, rest assured without tweaking the sampler!


Benchmarking randomness: this is brought by the fact a CUDA convolution operation usually has several implementations, and each might result in slightly different values due to the approximation they might use. So if benchmark option is True, our application might end up choosing different implementations across runs according to the state of our machine at that time, which leads to possible discrepancy.

To disable this part of randomness we need this line:
torch.backends.cudnn.benchmark = False



CUDA algorithmic randomness: even if we have fixed our choice of CUDA implementation of the operations, the implementation itself might have randomness. I’m not familiar with the exact algorithm but it seems some algorithms would rely on certain sampling strategies or approximation, somewhat like the application-level randomness but we will not be able to touch it. This part includes:


CUDA convolution randomness that happens in CUDA convolutions, and can be disabled by setting torch.backends.cudnn.deterministic = True; and

other operations’ randomness such as torch.bmm() (full list here), and can be disabled by calling torch.set_deterministic(True). However, this also takes care of the CUDA convolution randomness, so it is enough to use this line to turn off all CUDA algorithmic randomness.



To summarize, we would need the following lines to make completely reproducible (not across Pytorch versions & platforms, only our own runs on fixed machines) experiments:
# Application-side randomness
np.random.seed(seed)
torch.manual_seed(seed)
random.seed(seed)

# Benchmarking randomness
torch.backends.cudnn.benchmark = False

# CUDA algorithmic randomness
torch.set_deterministic(True)

I still want to make two notes here in case people get confused:


torch.set_deterministic(True) shuts off algorithmic randomness for all CUDA operations, whereas torch.backends.cudnn.deterministic = True only shuts off the CUDA convolution randomness. So it is enough to just use the former one.


torch.backends.cudnn.deterministic and torch.backends.cudnn.benchmark mean totally different things:


...deterministic = True disables the CUDA convolution randomness mentioned above, whereas

...benchmark = False disables the algorithm selection.



And, as @mmisiur suggested, the Reproducibility Doc 3 is always a good reference.