You mean “why does so much work go into keeping tensors on or off the GPU”? Memory limitations are the obvious one. The ideal goal for any GPU training is that you find yourself bound only by the processing speed of the GPU: every batch is loaded and transferred to the GPU in time for training the next epoch and as much VRAM is used as possible (for larger batch-sizing, or with smaller batches to prefetch more). Accessing anything that’s on the GPU from the CPU is either impossible or requires syncing and transferring (depending on what the method is), so that’s something to avoid too.
Fortunately, the newbie friendly option is the data loader rather than writing individual batches, then you just need to make sure the model is running on the GPU.