Thank you for your comment. I tried to use WeightDrop  in torchnlp package as well. However, it returned the same error. Since both codes depend on _weight_drop(), I believe that something goes wrong in that function when I wrap my model by torch.nn.DataParallel.
I understand that DataParallel split the input into inputs[] (list of parts of inputs), and inputs[i] is on the i-th GPU. DataParallel also replicates my model wrapped by torch.nn.DataParallel to replicas[] (list of copy models), and replicas[i] is on the i-th GPU. In my case, replicas1 returns runtime same error. the reason for this error is that replicas[1].weight and replicas[1].bias are not properly assigned in the 1st GPU. Could you have any idea for this? Manually assigning such tensors to the target GPU is unstable as you pointed out (I tried, but it returns same error for some input).