I would claim it all depends on your use case and there is no silver bullet.
By default you would want to use the GPU to train the model and the CPU to load and process the data. Using multiple workers in the DataLoader the data loading and processing will be applied in the background, while the GPU is busy with training the model.
torchvision.transforms are using PIL as the default backend for image transformations, which uses numpy arrays internally.
However, if you need differentiable transformations (e.g. via Kornia), you would have to apply them on tensors and cannot use numpy anymore (or would need to write the backward functions manually).
Also, on “bigger” systems, you might see that the CPU is not fast enough feeding the GPUs, which is visible in starving GPUs.
In that case you could use e.g. NVIDIA DALI 3 to apply the transformations on the GPU, which could give you an overall performance boost.