So you what you need is leaving all the other submodules but a specific module at a time to be tracked by autograd? Couldn’t it be achieved by freezing/undoing parameters taking turns or use detach() properly inside your forward loops?
Or if possible, autograd provides local disabling of gradient computation 1 could be of your use. (I haven’t used this functionality thus not so sure about tho)