I strongly advise you to read Pytorch reproducibility 3
In my case what helped was not only setting torch and numpy seed but also setting cudnn.benchmark to false and torch.set_deterministic:
torch.manual_seed(seed)
np.random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.set_deterministic(True)

But there is a chance that it will be not enough. In that case follow Pytorch reproducibility 3
Hope that helps!