Solved by ptrblck in post #4 
              
              
                I used the ImageNet example and set one target value to 1001 using a single GPU and DDP. 
In both cases it crashed with the expected error message. After using ignore_index=1001 both runs passed.