I tested all cases:
res2, res3, res4, res5 = checkpoint.checkpoint(self.resnet_backbone, data['data'])
print(res2.requires_grad, res3.requires_grad, res4.requires_grad, res5.requires_grad)
res2, res3, res4, res5 = checkpoint.checkpoint(self.resnet_backbone, data['data'].requires_grad_())
print(res2.requires_grad, res3.requires_grad, res4.requires_grad, res5.requires_grad)
res2, res3, res4, res5 = self.resnet_backbone(data['data'])
print(res2.requires_grad, res3.requires_grad, res4.requires_grad, res5.requires_grad)

This outputs:
False False False False
True True True True
False True True True

In the original resnet backbone architecture of my code I have frozen res2 layer, could this be the issue?
Edit: it appears this is the culprit as I can train when I unfreeze the res2 layer. I suppose to fix this I can just .detach() it, ie.
res2, res3, res4, res5 = checkpoint.checkpoint(self.resnet_backbone, data['data'].requires_grad_())
res2 = res2.detach()