I got the same problem.
My model was trained on 10-fold cross validation. I tried to save all the random states that may contribute to reproducibility every one fold was trained, the state including:
    state = {
        'fold_iter_idx': fold_iter_idx,
        'torch_rng_state': torch.get_rng_state(),
        'cuda_rng_state': torch.cuda.get_rng_state(),
        'np_rng_state': np.random.get_state(),
        'random_rng_state': random.getstate()
    }

Then a new model was initialized to train on next fold. Finally there are 10 results respect with to 10 folds.
I can reproduce same results if I start the program from the very beginning (from the first fold).
But when I tried to reproduce the 7th fold result only with loading the 6th state that saved during previous training, the result was different from that got during previous training.
I tried to disable CUDNN and set
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

I didn’t manage to reproduce it.
But the results  with loading 6th states are same,
and I can reproduce the same result as the the results got in from the very beginning training  by loading states on CPU.
I am not sure whether I make my issue clear.