Dear All,
I would like to ask for your help! I’ve implemented a model described in this 6 paper but when I test the model I see strange results. That is the loss is always a really small number but the accuracy of the model on the training set is bad and during training it always takes on the same value.
Please find below an example:
Epoch:  1     Loss: 0.00005, Accuracy: 0.36782 Time: 3.7852s
Epoch:  2     Loss: 0.00005, Accuracy: 0.36782 Time: 3.6561s
Epoch:  3     Loss: 0.00005, Accuracy: 0.36782 Time: 3.6510s
…
I know that there might be errors in the code that calculates the accuracy but I would like to ask you to have a look at the implementation of the loss function. I’m fairly new to PyTorch which is why I’m not sure if I’ve done everything correctly. Please find the code snippet below:
class EnergyLoss(nn.Module):
    """
    Implements the loss function described in
        paper:
            'Learning Text Similarity with Siamese Recurrent Networks'
        by:
            Paul Neculoiu, Maarten Versteegh and Mihai Rotaru
            {    neculoiu,         versteegh,          rotaru}@textkernel.nl
    """

    def __init__(self, margin):
        """
        Initializes the loss function
        :param margin:
        """
        super(EnergyLoss, self).__init__()
        self.m = margin

    def forward(self, cos_sim, y_target):
        """
        Calculate loss
        Dimensionality:
        N: number of examples in the current minibatch

        :param cos_sim: vector of cosine-similarities which is of shape (N, 1)
        :param y_target: binary vector of target classes which is of shape (N, 1) [1: similar; 0: dissimilar]
        :return: energy_loss: loss calculated on the current minibatch
        """
        # input check
        if len(cos_sim.size()) != 2 or cos_sim.size()[1] != 1:
            raise ValueError("The vector of cosine similarities is invalid! Size: (N, 1)")
        if len(y_target.size()) != 2 or y_target.size()[1] != 1:
            raise ValueError("The vector of target classes is invalid! Size: (N,1)")
        # minibatch info
        nb_examples = cos_sim.size()[0]
        # loss calculation
        energy_vec_pos = torch.zeros((nb_examples, 1)).type(float_type)
        energy_vec_neg = torch.zeros((nb_examples, 1)).type(float_type)
        # for filling the output vector
        pos_mask = y_target == 1
        neg_mask = y_target == 0
        # get corresponding entries
        pos_sim = cos_sim[pos_mask].type(float_type)
        neg_sim = cos_sim[neg_mask].type(float_type)
        # apply defined functions
        # similar case
        pos_sim = 0.25 * (1. - pos_sim) ** 2
        # dissimilar case
        neg_sim[neg_sim >= self.m] = 0
        neg_sim[neg_sim < self.m] = neg_sim ** 2
        # create vectors for both cases
        # energy_vec_pos[pos_mask] = pos_sim
        energy_vec_pos = Variable(
            energy_vec_pos.masked_scatter_(pos_mask.data, pos_sim.data),
            requires_grad = True
        )
        energy_vec_neg = Variable(
            energy_vec_neg.masked_scatter_(neg_mask.data, neg_sim.data),
            requires_grad = True
        )
        energy_loss = torch.sum(
            y_target.float() * energy_vec_pos.float() + (1. - y_target.float()) * energy_vec_neg.float(),
            dim = 0
        )
        return energy_loss

Thank you very much for your help in advance!