I don’t know how your workload is supposed to work exactly, but based on the description it seems that you would like to train multiple models and load the most recent checkpoint?
If so, it seems that you are using different models which might be causing the shape mismatch error. In that case I would assume that you would like to store the checkpoints for each model separately using a different name or folder.