Hey, sorry for the late reply.
Indeed I found a post of yours about model ensemble somewhere else, and decided to go with creating 3 instances of the same model, load all their state dictionaries in the usual way:
define 3 submodels
model0 = PABN()
model1 = PABN()
model2 = PABN()
# load state dict of 3 models
   checkpoint = torch.load('epoch_500.pth')
model0.load_state_dict(checkpoint['model_state_dict'], strict=False)
model1.load_state_dict(checkpoint['model_state_dict'], strict=False)
model2.load_state_dict(checkpoint['model_state_dict'], strict=False)
# freeze backbones
freeze_backbones = True
if freeze_backbones:
    with torch.no_grad():
        # freeze parameters
        for name, child in model0.named_children():
            if name == 'backbone':
                for name2, params in child.named_parameters():
                    params.requires_grad = False
        for name, child in model1.named_children():
            if name == 'backbone':
                for name2, params in child.named_parameters():
                    params.requires_grad = False
        for name, child in model2.named_children():
            if name == 'backbone':
                for name2, params in child.named_parameters():
                    params.requires_grad = False
# create a new model ensemble
model = EnsembleModels(model0, model1, model2)

This ended up in just as it would be an actual branching tree network. There are 3 different backbones, but they are frozen, so they do not change and stay exactly the same. The downside is the higher memory usage I guess.

Note that if i did not freeze the backbone i got a lower performance of the ensemble model.
But currently I would like to further improve my architecture and branch my network another time:

image472×629 5.01 KB

And here I have the problem again. And this time I would like it again if i could copy weights and biases in a reliable way, but i still dont know how to do it. When I used the previously mentioned methods (copy weigth data + bias data), i got a close but still not same performance, therefore i cannot trust it, and the training after that resulted in lower performance as I remember, compared to the case of loading with state dictionary and ensembling with frozen conv0-3.
Since I know that training the conv4+conv5 resulted in better perfomance, Im not sure about freezing conv4 as well, as I have the feeling that those higher level features do need some re-trainign in order to specialize for their data. So I cannot really use the same method, when i froze the conv0-3. I think freezing conv0-3 was okay because they have low level feratures that doesnt really need specialization too much.