Thank you for your answer! I tried DistributedDataParallel and the performance finally makes sense. At least the simplest model(depthwise) can train and inference faster than the two others(dept+pointwise & naive) on multiple GPUs. Here is the result for training:
====== start test! gpu num: 1 ======
1 gpu depthwise conv total time: 2.508331537246704 for 100 iterations
1 gpu dept+pointwise conv total time: 9.373668670654297 for 100 iterations
1 gpu naive conv total time: 8.2487051486969 for 100 iterations
====== start test! gpu num: 4 ======
4 gpu depthwise conv total time: 12.313549995422363 for 25 iterations
4 gpu dept+pointwise conv total time: 13.747057676315308 for 25 iterations
4 gpu naive conv total time: 13.246102333068848 for 25 iterations

Here is the result for inference:
====== start test! gpu num: 1 ======
1 gpu depthwise conv total time: 0.9100768566131592 for 100 iterations
1 gpu dept+pointwise conv total time: 4.834381341934204 for 100 iterations
1 gpu naive conv total time: 4.099367618560791 for 100 iterations
====== start test! gpu num: 4 ======
4 gpu depthwise conv total time: 7.846449375152588 for 25 iterations
4 gpu dept+pointwise conv total time: 8.739933252334595 for 25 iterations
4 gpu naive conv total time: 8.57079792022705 for 25 iterations

As for the overhead in communication, I will embed these models to a larger model and see what happens.