@ptrblck okay, which is the good way?

split the layers of pretrained model and concatinate with the layers of my custom model (according the layers output size which are required for fcn), then train the custom model.
use pretrained weights as features (remove final layers which are not required  and custom classifier layers and then train.
for example in the second method i used vgg features,

class fcn(nn.Module):
def init(self):
super(fcn, self).init()
self.features = vgg16.features
self.classifier = nn.Sequential(nn.Conv2d(512, 4096, 7),nn.ReLU(inplace=True),nn.Conv2d(4096, 4096, 1),nn.ReLU(inplace=True),nn.Conv2d(4096, 32, 1),nn.ConvTranspose2d(32, 32, 224, stride=32)
)
def forward(self, x):
x = self.features(x)
x = self.classifier(x)
#print(x.shape)
return x
Also, in both case, what is the affect of freezing the features. Should i train it or not?
for param in vgg16.features.parameters():
param.requires_grad = False
Thank you