I had successfully staggered model training, saving state after every epoch using the standard PyTorch method (saving state_dict, optimizer state and so on to a dictionary and serializing), for separate Encoder and Decoder objects.
However, when incorporating these separate objects as layers in an aggregate ‘Model’ object, and trying to save and load its state after a stretch of training, my loss figures in the next epoch go back to as though I started training from scratch.
I am doing everything as before except that I have nested PyTorch modules, and train the model which initialises and incorporates the encoder and decoder.
Am I incorrect in thinking I should be able to nest modules in this way and save/load the state of the module which contains the others to resume training?
What’s especially odd is that training is working with this setup, but just won’t resume.