I had planned on using depthwise convolutions on a future network, so was curious about this as well.
I forked that code and made the benchmark a little more extensive at https://github.com/DNGros/pytorch-mobilenet/blob/master/benchmark.py 26 to cover different batch sizes, run with and without CUDA, and do multiple trials rather than just one.
Pytorch v1.0.0
GPU name GeForce GTX 745
-----CUDA True-----
--batch size = 1
resnet18        0.05661s
alexnet         0.01414s
vgg16           0.01525s
/usr/local/lib/python3.6/dist-packages/torchvision/models/squeezenet.py:94: UserWarning: nn.init.kaiming_uniform is now deprecated in favor of nn.init.kaiming_uniform_.
  init.kaiming_uniform(m.weight.data)
/usr/local/lib/python3.6/dist-packages/torchvision/models/squeezenet.py:92: UserWarning: nn.init.normal is now deprecated in favor of nn.init.normal_.
  init.normal(m.weight.data, mean=0.0, std=0.01)
squeezenet1_0   0.02958s
mobilenet       0.04543s
mobilenet one group 0.10287s
mobilenet four group 0.11901s
--batch size = 4
resnet18        0.03556s
alexnet         0.00807s
vgg16           0.01727s
squeezenet1_0   0.04847s
mobilenet       0.06989s
mobilenet one group 0.29338s
mobilenet four group 0.28810s
--batch size = 32
resnet18        0.18783s
alexnet         0.00866s
vgg16           0.02191s
squeezenet1_0   0.38219s
mobilenet       0.53036s
mobilenet one group 2.59376s
mobilenet four group 2.26333s
-----CUDA False-----
--batch size = 1
resnet18        1.14055s
alexnet         0.37858s
vgg16           2.17955s
squeezenet1_0   0.46393s
mobilenet       1.27138s
mobilenet one group 1.59579s
mobilenet four group 1.26217s
--batch size = 4
resnet18        2.22609s
alexnet         0.66891s
vgg16           6.26908s
squeezenet1_0   1.56296s
mobilenet       2.62577s
mobilenet one group 3.63102s
mobilenet four group 2.69251s
--batch size = 32
resnet18        12.45708s
alexnet         2.78703s
vgg16           45.08901s
squeezenet1_0   9.72596s
mobilenet       15.53032s
mobilenet one group 22.75712s
mobilenet four group 17.14482s

Note, this on a pretty old GPU (GTX 745) and CUDA (8.0). You might want to try running on this as well and see what you get on your machine.
I definitely see an improvement when groups=input_channels compared to one group, but is at best maybe 5x for larger batch sizes on GPU, and only maybe a 1.5x improvement on CPU. That speedup certainly respectable, but I somewhat expected a larger speedup given the much greater reduction in params and ops depthwise convolutions should have given. Not sure if this is expected and matching that of other frameworks, or if this is a pytorch issue.