The last non-linearity is F.tanh()
the details of criterion I use are attached below
 def forward(self, input, target, mask):
        # truncate to the same size
        # input, target, mask like (60, 17, 2631), (60, 17), (60, 17)
        target = target[:, :input.size(1)]
        mask =  mask[:, :input.size(1)]
        """
        batch_size = target.size(0)
        seq_length = target.size(1)
        input = input.view(batch_size * seq_length, -1)
        target = target.view(batch_size * seq_length)
        # mask = mask.view(batch_size)
        output = self.loss(input, target)
        output = output.view(batch_size, seq_length)
        output = output * mask
        """
        # pick the dims correspond to words
        output = -input.gather(2, target.unsqueeze(2)).squeeze(2) * mask # like (60, 17)
        output = torch.sum(output) / torch.sum(mask) # scalar, average log likelihoods across all time steps of all captions in a batch, positive values 
        return output

I both tried to use nn.NLLLoss or use gather to calculate loss by myself, but neither of them works. The input of this function is after F.log_softmax().