Eiphodos:

In the checkpoints for modelA and modelB I also saved the amp state dict. Should those be loaded in this case and how?


This wouldn’t be necessary, if you are not planning to finetune the pretrained models.



 Eiphodos:

Does amp.initialize support models embedded inside an ensemble?


We recommend to use the native amp implementation via torch.cuda.amp instead of apex/amp. Since there is no amp.initialize method, it should just work, but let us know if you encounter any issues.



 Eiphodos:

What if I want to unfreeze the lower models at a later point, do I need to reinitialize both the optimizer and amp in that case?


The amp state_dict stores the loss scaler values etc. which is useful to continue the training with the same setup. If you initialize a new scaler, the first iterations might be skipped, if the scaling factor is too high, which wouldn’t necessarily be the case using the scaling factor from the last iteration in the pretraining.