I read all the posts regarding reproducibility including https://pytorch.org/docs/stable/notes/randomness.html 22. Here are my experiences so far:


calling the below code segments makes sure that I got consistent results within the same machine.
seed = 1234
random.seed(seed)
os.environ[‘PYTHONHASHSEED’] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False


My goal is to get consistent results over different machines. For this, I made sure that I have the same python, numpy, pytorch, torchvision versions etc. for both machines. Still, results are very different for two machines.


To point out which part of the code results in inconsistency, I removed the parts of the code one by one
i- One reason was dropout. When I change to in_replace dropout, this problem solved. I.e. dropout gives consistent results.
ii- The only remeaning part that leads to inconsistent results is optimizer.step(). However, I could not find any solution to solve this inconsistency.


I am not sure if loss.backward() or optimizer.step() functions use CUDA functions mentioned in https://pytorch.org/docs/stable/notes/randomness.html 22. If so, can anyone suggest a possible solution for inconsistencies due to optimizer.step()?
This is very important for me. Even if there is no absolute solution to this problem, I like to try different options.