I would recommend to create tensors on the CPU in your Dataset and push it (asynchronously) to the GPU in your training loop. If you are using multiple workers, each worker might try to initialize a CUDA context, which will yield an error.


Each worker will load a batch of your Dataset by calling __getitem__ with batch_size indices, which also means that all preprocessing will be performed in this process. You should experiment a bit with the number of workers, as the performance depends on the number of CPUs your system has.