To add onto this topic, it would be very helpful to know exactly how ignore_index works. For instance, consider:
N = 10
criterion = nn.CrossEntropyLoss(reduction='none', ignore_index=-1)
groundtruth = torch.rand(N, ).ge(0.5).type(torch.LongTensor)
groundtruth[7:] = -1
pred = torch.rand(N, 2)
loss = criterion(pred, groundtruth)

In such a situation, Pytorch returns a value of 0.0 for elements which are to be ignored. Running the same piece of code with N = 5000 returns weird numbers in the loss for elements to be ignored. Values such as .0646e+24 etc.
Next, how do we backprop on the above said loss?
Can I simply find all elements which aren’t -1, say, loc = groundtruth!=-1, and average by ignoring those values?
loss = torch.mean(loss[groundtruth!=-1])
loss.backward()

For some weird reason, the above mentioned situation does not work for me. The code crashes after 10 epochs or so.