Hi, I found this both interesting and crucial. I want to share my experience here as well and hope it is useful.
Let us assume we have a variable B. I have seen case where I explicitly set B.require_grad = False but B is still updated during fine-tuning. This could happen even when I put operations involved with B within torch.no_grad().
My solution is to modify the code declaration of B in the code. Something like this:

self.register_buffer(“B”, torch.ones(the shape of B looks like), requires_grad=False))

With that when I load a pretrained model I noticed B is loaded properly. More importantly, when I fine-tune the model I noticed B is indeed frozen. This is a bit manual but it works.