To speed up the transfer of data from the host to your device you could use pin_memory=True. However, if your data already is on the GPU you don’t need this function and it will throw an error.
If you have enough GPU memory to hold the data and model this is surely a valid approach to save the loading overhead.