Solved by googlebot in post #2 
              
              
                Your input-to-hidden layer should use 1 group (but bigger output width), otherwise you split inputs. 
As I side node, I just switched from this approach to batched matrix multiplications, as there are some performance issues with grouped convolutions in cudnn. 
I also found that creating block-diago…