Ok, one follow up question. If I am to pretrain the model on one data set, save the weights and optimizer for continuing to train later, once I load in these weights to train on the new data set, should I reset the optimizer when I start to train on the new dataset? or should I load in the optimizer dictionary information? I was looking at a pytorch documentation about saving models and it said to save the optimizer when you want to continue training, but I am not sure if that applies to pretraining and then training on a new dataset.