I’m sorry I don’t understand this.  autograd.grad  is does forward differentiation, doesn’t it?

No it does backward mode as well. Only that instead of populating the .grad field of all the leaf Tensor, it computes and returns the gradients for a list of Tensor that is given.

My thinking was that checkpoint removes all requires_grad by design

No the output of the checkpoint module should require gradients. If you put the whole network into it, you might need to set requires_grad=True on the input though. Does it help?