I am training a Siamese Network using Resnet50 pre-trained network. I have modified the fully connected layer at the end with a 3 layer classifier and frozen all of the resnet layers before the adaptive average pooling.
The loss function I use is below:
class TripletLoss(nn.Module):

def __init__(self, embedding_size):
    super(TripletLoss, self).__init__()
    self.margin = embedding_size

def forward(self, anchor, positive, negative):
    dp = (anchor - positive).pow(2).sum(1)  
    dn = (anchor - negative).pow(2).sum(1)  
    losses = F.relu(dp - dn + self.margin)
    acc =  (losses == 0.0).type(torch.float32).sum()

    return losses.sum(), acc

The network is trained on 1400 triplets, and 150 triplets are used for validation.
Below is the plot for the loss and accuracy. I am very confused about how to interpret this as the validation loss increases after epoch 5 but the validation accuracy either remains the same or increases slightly. Would this be a good model as the accuracy is high? Or is something wrong?
image862×307 28.5 KB
Any comments or suggestions to improve training would be appreciated.