We are trying to implement segnet which has encoder weights from vgg 16.
So, we created new cnn layers and are loading the weights of encoder from corresponding vgg layer.  Vgg16 is used only for weight initialisation. The encoder weights would be trainable after this.  Vgg is not part of forward call otherwise.