I am doing image classification where I have two input networks, of which the output nodes are concatenated in a fully connected layer. This is my code:
class resnet18_meteo(nn.Module):
    def __init__(self, resnet18, meteo_NN, num_classes):
        super(resnet18_meteo, self).__init__()
        
        # Respectively a torchvision resnet-18 and a 1-hidden layer NN
        self.resnet_CNN = resnet18
        self.meteo_net = meteo_NN
        
        # Sizes of the FC layers of both NN's
        self.len_fc_resnet = self.resnet_CNN.fc.in_features
        self.len_fc_meteo = self.meteo_net.fc_last.out_features
        print(self.len_fc_meteo)
        
        # Remove FC layer from the resnet 
        self.modules=list(self.resnet_CNN.children())[:-1]
        self.resnet18_convblocks= nn.Sequential(*self.modules)
        
        # Fully connected layer is now size resnet FC + meteo FC
        self.fc = nn.Linear(self.len_fc_resnet + self.len_fc_meteo, num_classes)
    
    def forward(self, img_x, meteo_x):
        
        # Both should be flattened layers at end of networks
        img_x = self.resnet18_convblocks(img_x)
        meteo_x = self.meteo_net(meteo_x)
        
        # Flatten convolutional features
        img_x_flattened = img_x.view(img_x.size(0), -1)
        
        # Concat the outputs of CNN and meteo-NN in fully connected layer
        out = torch.cat([img_x_flattened, meteo_x], dim=1)
        out = self.fc(out)
        return out   

I use a resnet 18 structure to input the images. Besides, I have four additional non-image features that  I input to a small feedforward network. As can be seen, the outputs of these networks (size 512 + 32) are thus concatenated in the FC layer.
Now I was wondering if I can give some kind of ‘judgement weights’ to these networks. For instance, I’d like to have the resnet have 70% of the influence on the decision/loss, and the non-image net have 30% of the influence.
The only way I can think of doing this is by totally disconnecting these networks from each other, calculate the losses for each data-point (for each network), and then weighting these losses. However, I would like to know if this type of thing could be incorporated into my current network structure.