Your input-to-hidden layer should use 1 group (but bigger output width), otherwise you split inputs.
As I side node, I just switched from this approach to batched matrix multiplications, as there are some performance issues with grouped convolutions in cudnn.
I also found that creating block-diagonal weight matrices can be better that bmm, as the latter creates an expanded copy of the weight tensor. I believe the best approach is shape (or sparsity) dependent.