The problem is that the actual model is much more complicated than this. sub1's inputs and outputs are in very different formats from my_module. I have to train sub1 with its direct outputs.
The process looks like,

train my_module in a separate script with a different loss function
freeze sub1, get output from sub1, out1
sub2 still requires_grad, feed out1 into sub2 to get out2
sub1 still freezing, feed out2 into sub1, but not using sub1’s __call__ because of input format issue, to get out3.
out3 and out1 are compared to get a loss.
In this process, not requiring grad is probably just a partial fix because I have to let go of __call__, which may mess up the autograd mechanism.