It should be much faster to use depthwise convolutions if I’m implementing it properly. See this GitHub issue 86 for example. Any time the number of groups is set equal to the number of input channels, that layer executes 10-100x faster. That should be apparent even when using a simple timing mechanism such as time.time().
Note that in the above link you’re looking for any lines that say Group=1024, since that was the size of their input.