At the moment, there isn’t a way to speed up group convolutions (from the code, this is what seems to being done).
The next version of CuDNN hopefully has this implemented. (generally Convolutions on the GPU are notoriously hard to speedup because we’re already at a place where CuDNN does a ridiculous good job).