If you are using multiple workers in your DataLoader, multiple initializations of the CUDA context might yield this error.
Try to use the spawn method for multiprocessing as described here 43 or alternatively use a single worker (num_workers=0).