Hello,
As the title states, I have a question on the behavior of the torch dataloader when I resume training process from the existing checkpoint.
My codes for loading dataset and dataloader looks like the following. (I’m using Huggingface’s datasets.)
train_dataset = load_dataset(args.dataset_loading_script_path,
                                     data_files=args.dataset_txt,
                                     split='train')
train_dataset = train_dataset.map(lambda examples: tokenizer(examples['sent_1'],
   max_length=args.max_length, padding='max_length', truncation=True), batched=True)
train_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask',])
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=total_train_batch_size, shuffle=True)

My question is, when I load the existing checkpoint and optimizer and resume training, is there a way to avoid training on batches (or training examples) that I’ve already trained on?
I’m assuming torch dataloader randomly shuffles data and feeds batches into the model each time the training starts and just simply skips the number of trained steps when iterating through the train dataset using dataloader.
If this is the case, will I be able to achieve what I’m trying to do by shuffling dataset before feeding it into the dataloader (and save dataset to reuse) and drop shuffle=True option?