I think I’ve found the workaround. When we do .cuda(), we may specify the GPU device we want to load data or model to make sure they are on the same GPU.  For example,
net = Net()
net = net.cuda( 0 )

input = Variable(torch.randn(1, 1, 32, 32))
input = input.cuda( 0 )
output = net(input)