Answering my own question:
dtype was a cuda type, and I think what was happening was multiple CUDA tensors was created in the dataset, and pytorch does not support this kind of behavior.
Switching it back to torch.FloatTensor, or not calling cuda type tensors will not cause the error.