I cannot reproduce this issue with this small code snippet:
x = torch.randn(10, 19, requires_grad=True, device='cuda')
y = torch.randint(0, 19, (10,), device='cuda')
y[0] = 255

criterion = nn.CrossEntropyLoss(ignore_index=255)
loss = criterion(x, y)
print(loss)

Are you sure that the ignored index 255 is causing the issue and not another unexpected target index?