Are you preloading the data (and target) tensors onto the GPU before passing them to Dataset?
This might cause multiple initializations of the CUDA context, which might yield this error.
You could try to set num_workers=0 in your DataLoader, use CPU tensors, or try to set the spawn method for multiprocessing.