Now you need to make sure that all your datas are on the same device.
Be careful though that doing y = x.cuda() is an autograd operation and so will cause the same issue of doing backward twice if you use y in your forward pass while x requires gradients.