Ah I didn’t think to check that the parameters are changing, and it appears they are due to the tuneable parameters after the checkpoint, but indeed this seems to be the case. If I add a line to print the change in a parameter (like a conv. weight) at each step, it is indeed zero when the checkpoint is on. And this explains everything that I observed. Thank you.
That is also strange because after pretraining using this my supervised learning did slightly better… but I suppose that may have been random chance because it was only slight (surprisingly slight!).
I guess I don’t fully understand how to use a checkpoint, then. Any chance you could give a short example of it in action? When I looked for some I found another person asking for examples who never did: Do you have examples of the usage of torch.utils.checkpoint 4