Ok wait the error in your code is happening because x1 and x2 are not initialized to cuda. You just define them like this
x1, x2 = torch.randn(64,3,3,3), torch.randn(1, 64)

you need to put those on cuda.