Hi,
I have been observing some memory issues for quite a while. Whenever I resume training from a checkpoint more GPU memory is required than when running from scratch. I have observed this independent of architecture/application (NLP, Vision). It’s hard to quantify because it depends on the model, but I’d guess it is around 5-10%.  Consequentially, I have to reduce the batch size in order to continue training from a checkpoint, which is a bit painful.
Any idea?