Perhaps, this looks to me like a use case for using nn.functional APIs. I mean,  you can write C network fully interms of functional APIs. A bit more effort though. But you could get away from spending effort to juggle with requires_grad stuffs.
P. S.  I hope I understood the question correctly. Pardon me if I’m way too off from the original problem:)