Thank you all, I got it by specialized the Gather function.
Btw, I wonder why replicate function will be iterative call when DataParallel Module forward rather than call once when init? https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L67 15