This is usually a label problem. Can you do the following:

save logits.shape and labels.cpu() in temporary variables,
after the device assert is triggered, print these.
If you have a label that is >= num_lables or < 0, you have the problem.