You can take out the model state dict from your encoder part and add then take out the other part from the ResNet model.
Encoder = encoder.__dict__

you can then take out the first layer like:
first_layer = Encoder._modules['layer1']

and then you can either take out the layers that you need from your resnet or you can just change the layers.
resnet = torchvision.models.__dict__(resnet50)(pretrained=true)
final_layer = resnet._modules['layer4']

and then pass combine these two. Hope this helps