Hi,
I am attempting to fuse features from colour and visible images using a pre-trained ResNet50 model. It works fine when i train it on colour images and provides decent results, however, when I try to use it in a way to fuse the colour image with thermal images within the network, the model trains fine but then the loss gets stuck around 20%.
I am using matlab to calculate the miss rate, which is  quite high, at about 50% and does not seem to decrease. I have tried to increasing/decreasing the learning rate but this not help much.
Based on papers i have read online on this topic, 6-10 epochs should be sufficent to train the model, however this has not been the case for me and i believe it has something to do with the way i am fusing the data.
Please find my fusion code below:
class DeepFusionNetwork(torch.nn.ModuleDict):
    def __init__(self, model):
        super(DeepFusionNetwork, self).__init__()
        self.model = model
        self.conv1 = self.model.body.conv1
        self.bn1 = self.model.body.bn1
        self.relu = self.model.body.relu
        self.maxpool = self.model.body.maxpool
        self.layer1 = self.model.body.layer1
        self.layer2 = self.model.body.layer2
        self.layer3 = self.model.body.layer3
        self.layer4 = self.model.body.layer4
        self.fpn = self.model.fpn
        self.conv2 = nn.Conv2d(1024, 512, kernel_size=(1,1), stride=(1,1), bias=False)

    def forward(self, img1, img2, targets=None):
        out = OrderedDict()
        self.conv2 = self.conv2.to(device)
        x, y = self.conv1(img1), self.conv1(img2)
        x, y = self.bn1(x), self.bn1(y)
        x, y = self.relu(x), self.relu(y)
        x, y = self.maxpool(x), self.maxpool(y)
        
        x, y = self.layer1(x), self.layer1(y)
        out[0] = x

        x, y = self.layer2(x), self.layer2(y)
        z = F.relu(torch.cat((x, y), dim=1))
        z = self.conv2(z)
        
        out[1] = z
        z = self.layer3(z)
        out[2] = z
        z = self.layer4(z)
        out[3] = z
        features = self.fpn(out)
        return features