Hi,
Many thanks for your reply!
when I changed .cuda() to .cuda(self.mu.device)  or .to(self.mu.device) It still raise RuntimeError: arguments are located on different GPUs.
        epsilon = self.normal.sample(self.mu.size()).to(self.mu.device)

Here are some details.
  File "/home/bzg/anaconda3/envs/torch1.2/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 1 on device 1.



    return F.linear(input, weight, self.bias)
  File "/home/bzg/anaconda3/envs/torch1.2/lib/python3.7/site-packages/torch/nn/functional.py", line 1371, in linear
    output = input.matmul(weight.t())
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorMathBlas.cu:260