Hi,
One thing to check with checkpointing is that at least one input to the module needs to require gradients. So if it is the first module in your net, and your input does not require gradient, it can prevent the first layer from training. Setting the input to require gradient is a simple workaround.

I notice that when I accumulate gradients there is consistently worse generalization

What do you mean by accumulate gradients? You mean using .backward() vs autograd.grad() ?