Thanks for your investigation, I saw the source code and I think you are right.
I will try your solution of two nested modules, can you give a bit more details about this solution?
I can also do something like:
def forward(self,x,flag):
    if flag:
       # do foraward1 code
    else:
       # do foraward2 code

And then:
        images = images.cuda()
        target = target.cuda()
        output = model(images,flag=True)
        output = model(images,flag=False)

Do you see any drawback to do the above path?