I meet a problem about reproducibility of pytorch.
I run a network multiple times, but obtain different result each time.
Here is my setting:
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
np.random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

I expect that the output of network should be the same every time I run the network, at least for the first few mini-batches.
But this is not the case, here are the experiment results(I choose the same gpu and seed each time, batchsize=1):

image821×841 76 KB

For random initialization case, loss of batch 1 is the same each time. It indicates that the initial weights are the same. However, if    using Adam, the result becomes different since batch 2.
For resuming from checkpoint case(optimizer is also restored), it produces expected result.
I am confused, I don’t find any random variable in Adam, and I know how to explain the result. Is there anything I didn’t notice when initializing Adam?
Any help is appreciated.