I cannot reproduce the issue using a single device as well as DDP.
If I pass ignore_index=255 to the criterion, the code runs fine.
Could you post an executable code snippet to reproduce this issue, please?