Solved by ptrblck in post #2 
              
              
                I would claim it all depends on your use case and there is no silver bullet. 
By default you would want to use the GPU to train the model and the CPU to load and process the data. Using multiple workers in the DataLoader the data loading and processing will be applied in the background, while the GP…