If you are preloading the data in your Dataset, you could directly push it to the GPU, e.g. in the __init__ method.
Usually you are lazily loading and processing each sample, which means you would have to push each sample to the device.