Guys, I have narrowed down the problem. The problem arises in the Aux Logits layer that is present in the Inception V3 architecture. Which on further inspection made me realise that I cannot think of inception model as a plain sequential model and I will have to in fact write my own module.
Is there any work around? Can I just add some layers (the way I intend to in the code) without the hassle of actually defining the model again?