Put the same code(model and training) and the same parameters on two GPUs(two same GPUs on one server).
But the different results appear. Why?
In the code, we set the random seed using the following code:
CUDA = torch.cuda.is_available()
import random
random.seed(123)
np.random.seed(123)
torch.manual_seed(123)
if CUDA:
torch.cuda.manual_seed(123)
torch.cuda.manual_seed_all(123)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
How to reproduce the experiment?
Here, the two figures from tensorboard as the following show different results.

Screenshot%20from%202019-11-21%2009-53-39682×647 41.3 KB
 
Screenshot%20from%202019-11-21%2009-59-54696×639 37.3 KB