Sure, just push the data to your GPU in your Dataset and use num_workers=0, since as far as I know multiple workers would otherwise try to initialize cuda multiple times which will yield an error.
If you are lazily loading your data in __getitem__ you would need to change this behavior and load all your data in __init__ of your Dataset or beforehand.