Hi everyone, recently I started to read about Siamese Nets and I wanted to try this type of model on a gender recognition task. My data is a .csv dataset containing ~3000 n-vectors of audio features (n=20). The loss function I’m using is the Contrastive Loss 9. Here is my model:
class ContrastiveLoss(torch.nn.Module):
  """
  Contrastive loss function.
  Based on: http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
  """
  
  def __init__(self, margin=1.0):
      super(ContrastiveLoss, self).__init__()
      self.margin = margin

  def forward(self, output1, output2, label):
      euclidean_distance = F.pairwise_distance(output1, output2)
      loss_contrastive = torch.mean((1-label) * torch.pow(euclidean_distance.double(), 2) +
                                    (label) * torch.pow(torch.clamp(self.margin - euclidean_distance.double(), min=0.0), 2))


      return loss_contrastive

class SiameseMLP(nn.Module):
  def __init__(self):
    super(SiameseMLP, self).__init__()
    self.layers = nn.Sequential(
        nn.Linear(20, 256),
        nn.ReLU(),
        nn.Dropout(p=0.1),
        nn.Linear(256, 256),
        nn.ReLU(),
        nn.Dropout(p=0.1),
        nn.Linear(256, 256),
        nn.ReLU(),
        nn.Dropout(p=0.1),
        nn.Linear(256, 2)
    )

  def forward_once(self, x):
    x = x.view(-1, 20)
    x = self.layers(x)
    return x

  def forward(self, x_1, x_2):
    y_1 = self.forward_once(x_1)
    y_2 = self.forward_once(x_2)
    return y_1, y_2

I started to do some tests and for now I can’t reach a good value for the training loss, keeping it around ~0.33. When I calculate the dissimilarity between pairs, I get random values so I think the model is not learning completely. Do you have any suggest? Thanks.