Great to hear that you resolved it yourself. 
ForwardHookManager in torchdistill does leverage the feature of forward hook in PyTorch like you did. It also takes care of managing handle produced by forward hooks for you, so it should be worth taking a look next time.