It may be unrelated. I think you could remove requires_grad=True, as you need not have this unless you really need gradient with respect to data.
What about the model? Do you use default pytorch modules or do you have any custom written modules?