If you start using PyTorch, you will notice that you will get errors if you have one tensor/array in cuda and the other on the CPU. Hence, it makes sense that if you are using cuda for some parts, it makes sense to use it for all parts.
There may be things that do not make sense to put into cuda. E.g,. say you load your image as a 2000x2000x3 and want to do some normalization and maybe other types of preprocessing. For the sake of the example, say you are interested in the 200x200 region only that you want to centercrop. Here, it makes most sense to just do that on the CPU via x[900:1100, 900:1100, :] before you push that cropped array onto the GPU instead of pushing the whole array onto the GPU and then cropping it there.