Thanks for the quick reply, actually i am a keras user trying to switch to pytorch.
I was using a keras model using Xception pretrained model
base_model0 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))
base_model1 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))
base_model2 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))
base_model3 = Xception(weights='imagenet',include_top=False,input_shape=(None,None,3))
x0 = base_model0.output
x0 = layers.GlobalAveragePooling2D()(x0)
x1 = base_model1.output
x1 = layers.GlobalAveragePooling2D()(x1)
x2 = base_model2.output
x2 = layers.GlobalAveragePooling2D()(x2)
x3 = base_model3.output
x3 = layers.GlobalAveragePooling2D()(x3)
x = layers.concatenate([x0,x1,x2,x3])
x = layers.Dense(4,activation='sigmoid')(x)
model = Model(inputs=(base_model0.input, base_model1.input, base_model2.input, 
    base_model3.input), outputs=x)

each Xception base model has (NONE, NONE,3) shape signifying any shape of images be inserted,
where NONE, NONE signifies height and width and ‘3’ the no. of channels.
then each result of the layers go through Global average pooling and are concatenated to give an output.
Although the input shapes were different but when it enters the Xception the output shape is same before going to the  Global average pooling .
I just wanted to know if I could implement with Pytorch.