I was having this same issue but on a tabular dataset using the fastai library on top of pytorch and couldn’t figure it out. Ultimately I fixed the issue by running CUDA_LAUNCH_BLOCKING=1 to get the real stack trace. Then I saw pytorch was trying to save to a tmp directory that was empty but for some reason caused the illegal memory access error, possibly because the directory was being used by something else in the background? Anyway deleting the empty directory fixed the issue.