This is a part of incremental learning, so I earlier dealt with FC layers and tried not to change the weights of conv_layer so as to generalize better but now to make it better I have to add more conv layers. So i want to use pretrained weights of resnet, and add two or more pathways after that. Each pathway would solve a task and rest would be turned off (freeze)
I want my forward pass to return a dictionary (clf_outputs) which would contain tensors (fc0,fc1) and feature vector.