Hello all. I’ve been trying to train a model with a dice loss function that has two inputs, “logits” and “true” with shapes of [4, 6, 256, 256] and [4, 256, 256] respectively. Unfortunately, the training process crashes with the following error:
IndexError: index 68788682753 is out of bounds for dimension 0 with size 6
and it crashes on this line of code in particular:
true_1_hot = torch.eye(num_classes)[true].cuda()

where num_classes = logits.shape[1]
The annoying thing about this error is that it isn’t consistent; in two tests I’ve done, the first time it trains for 5 epochs before crashing with this error, and in the next one it trains for 150 epochs before crashing. Therefore, I can’t figure out how it would go through this exact code multiple iterations of training without any issues, and then decide to crash all of a sudden. The index # it crashes on with that error always remains the same however.
Any help would be greatly appreciated!