Hi @ptrblck, thanks for confirming that for me. So, after the training and saving the checkpoint as follow:
## Save checkpoint ##
        PATH = 'checkpoints/'+str(index)+'.pth'
        torch.save({
            'epoch': index + 1,
            'valid_loss': valid_loss,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            }, PATH)

I should add optimizer.zero_grad() and optimizer.step() before loading the checkpoints, is that right?