Thanks for you reply.
However, I was not using DataParallel module as I was doing model parallelism instead. I distributed different parts of my model into different GPUs using torch.nn.Module.cuda and copied tensors between GPUs using torch.Tensor.cuda. I got that runtime error when I loaded my model on a single GPU.