Hi!
I followed the pytorch tutorial on the topic transfer learning but I am getting results that are clearly wrong. As the application I am trying to apply is different a had to make some tweaks and it would be great if someone can give me some insight.
You can see below that I am here removing the final layer of the pre-trained model instead of replacing it.
model = models.resnet50(pretrained=True)
model = torch.nn.Sequential(*(list(model.children())[:-1]))

And below you can see why I took a different approach. I need to compare the features of 3 images, produced by the pretrained model, and I do this by concatenating said features and passing it through my fully connected layers, which I will optimize.
class Net(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_dim, 600)
        self.fc2 = nn.Linear(600, 600)
        self.fc3 = nn.Linear(600, output_dim)

    def forward(self, input_1, input_2, input_3, pretrained_model):
        with torch.no_grad():
            output_1 = pretrained_model(input_1.float())
            output_2 = pretrained_model(input_2.float())
            output_3 = pretrained_model(input_3.float())

        s = torch.cat((output_1, output_2, output_3), dim=1).view(output_1.shape[0], -1)
        
        s = F.relu(self.fc1(s.float()))
        s = F.relu(self.fc2(s))
        output = self.fc3(s)
        return output

The “classification” process is based on whether image 2 or image 3 is more similar to image 1: prediction [0, 1] or [1, 0]
Using the Cross entropy loss and Adam optimizer the running loss does decrease, however when a pass the validation set, the model strictly speaking predicts the same thing for all possible inputs (since the data is evenly split between [0, 1] and [1, 0] I get an accuracy of 0.5).
Please let me know if you have any idea what might be going on here. Have been debugging for a long time now.