Suppose I’m using cross_entropy loss to do language modelling (to predict the next element in a sequence).
I have sequences with different lengths that I want to batch together, and the usual solution is to order them, pad with a special symbol (say 0), then use pack_padded_sequence(), feed them to an RNN and then .pad_packed_sequence().
However, for the loss function .cross_entropy() to ignore the padding, I should also use .cross_entropy(output_padded, ignore_index=0).
My question is – what happens if I do all of this, but without using pack_padded_sequence() and .pad_packed_sequence()? If .cross_entropy(output_padded, ignore_index=0) already ignores all 0’s, and assuming that 0’s only occur at the end of sequences, I don’t see how the outcome should differ.