Hi, I didn’t find any obvious solution for this problem since I had limited time at the moment and I decided acquiring the infrastructure for training in a single pass, however post the code on how you’re handling the checkpoint to see if it’s anything wrong, also try using the same random seed for all the training process, I guess that probably if you’re using any special sampler or operation that depends on randomness it could affect the performance if the seed is changed.
This is the code I use for seeding everything:
import numpy as np
import random
import os
def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
seed_everything(123)