What detach() is doing is detaching from previous history. In particular, you don’t want any history corresponding to your dataset because no gradients should be flowing back that way.
You convert to cuda before or after, that does not change much.
If you could get a small code sample to reproduce your problem, that would be very helpful.