Hi,
I am afraid I don’t have a very large practical experience. But for vision models, it is common to checkpoint the conv/relu/norm blocks. You can check here 16 how this is used for the torchvision models.