I’m not sure, why you don’t get a proper error message even with CUDA_LAUNCH_BLOCKING=1.
Usually I check that all batches, especially the targets, have valid values.
Often one unusual batch has e.g. a too high target values for whatever reason, so that NLLLoss will crash.
Since the error is thrown randomly, we could try to remove some random operations, e.g. random transformations, no shuffling, num_workers=0 etc. to narrow down the problem.
However, it’s also strange that the CPU code runs without any problems.
Can you think of some operation, which might mess up your targets or shapes of your input or target?