Good idea, I’m trying to get my runs to be deterministic at the moment, as suggested. I have:
random.seed(manual_seed)
np.random.seed(manual_seed)
torch.manual_seed(manual_seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

I’m not sure if it’s necessary, but I also set num_workers=0 to avoid possible indeterminism from multiple data loaders. The losses look very reproducible, but after 5 epochs the losses from multiple runs start to differ in the fourth decimal place, indicating that something is still a small indeterminism problem.
I looked at: https://pytorch.org/docs/stable/notes/randomness.html
This says:

A number of operations have backwards that use  atomicAdd , in particular  torch.nn.functional.embedding_bag() ,  torch.nn.functional.ctc_loss()  and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.

I use Conv2d with padding=1, BatchNorm2d, ReLU, ConvTranspose2d, and MaxPool2d. Are any of these candidates for unavoidable non-determinism?
I also ran with export PYTHONHASHSEED=0. What else could I be missing?