The current workaround would be to use a register_forward_hook() on the nn.Module as you were doing. And in this forward hook, add a Tensor register_hook() on the inputs (or outputs) that are of interest.
This looks very surprising. Can you try to get the smallest possible example that reproduces this so that we can open a bug report on github?