If I store them in the same file - it only stores the state_dicts for each. It will not pickle the object. Problems then if different “objects” have different models.
Separate files sounds troublesome. Which order do I load them in and do they still work for training.
Imagine one of a group of 10 model/optimizer/scheduler does particularly well after round one - where each has had an hour on the gpu.  This may be because I gave it a favourable set of hyper parameters.
I want to take this group, duplicate it x10, and then run each of those with slightly different hyperparameters.
In essence - I want to be able to duplicate the whole model/optimiser/scheduler state.