Hi,
It looks like you’ve got some indexing problems. Maybe one sample in your dataset has a wrong value for label? and it appears randomly depending on when this label is drawn by the dataloader?
Also cuda errors are asynchronous so if the error is raised within cuda, you can add CUDA_LAUNCH_BLOCKING=1 to make sure the error is raised at the right place.