yunjey:

Additionally, mask is multiplied by the calculated loss (vector not scalar) so that the padding does not affect the loss.


Then the non-averaged cross-entropy loss hasn’t come yet, but this could be easily implemented by torch.gather.
For example, suppose

the outputs is the logit for each label, whose shape is [batch_size, time_step, num_token](if not, you could always get them with F.log_softmax)

targets contains the targeted label for each step, whose shape is [batch_size, time_step]


Then the cross_entropy loss could calculated with
def nll_loss(outputs, targets):
    return torch.gather(outputs, 1, targets.view(-1,1))

Then you can apply masks as is in Tensorflow.