Thanks a lot for your help. I am sorry that you can not run it. I can run the code on my server.
I also guess the error is caused by self.lstm_c0 and self.lstm_h0. I tried the code you provided, but it still raises the runtimeerror: tensors are on different GPUs.

param = next(self.parameters())
is_cuda=param.is_cuda
if is_cuda:
cuda_device=param.get_device()
self.lstm_c0 = Variable(torch.zeros(self.nParams, self.lstm.hidden_size), requires_grad=False).cuda(cuda_device)
self.lstm_h0 = Variable(torch.zeros(self.nParams, self.lstm.hidden_size), requires_grad=False).cuda(cuda_device)