Ok, great. Thank you for clarifying! I didn’t realize torch.utils.checkpoint was used for tracing memory. It is a bit of a misnomer then. I think the first definition is consistent with other machine learning frameworks (at least with TensorFlow).
So is there a way to save and restore the mini-batch index along with the epoch index? I doubt it is as simple as:
 save_checkpoint({
            'epoch': epoch + 1,
            'mini_batch': minibatch + 1,
            ...
        }, is_best)

Is there functionality in the model or the optimizer to load a minibatch size and index alongside the epoch index?