Note that you may be comparing different implementations. With
m1 = torch.nn.Conv1d(256,256,3,groups=1, bias=False).cuda()
m2 = torch.nn.Conv1d(256,256,3,groups=256, bias=False).cuda()
a = torch.randn(1,256,5, device='cuda')
b1 = m1(a)
b2 = m2(a)

I get:
In: b1.grad_fn
Out: <SqueezeBackward1 at 0x7f0f35be90b8>
In: b2.grad_fn
Out: <SqueezeBackward1 at 0x7f0ed1ec2c50>
In: b2.grad_fn.next_functions
Out: ((<ThnnConvDepthwise2DBackward at 0x7f0ed1f007f0>, 0),)
In: b1.grad_fn.next_functions
Out: ((<CudnnConvolutionBackward at 0x7f0ed1ebf780>, 0),)

So you would be comparing the non-grouped CuDNN convolution with the “native” fallback TH(Cu)NN in the grouped case (which isn’t - or at least wasn’t - supported by CuDNN so PyTorch needs to fall back to it’s own implementation). Now I didn’t look in great detail at the Cuda THNN implementation, but when I ported libtorch to Android, the CPU THNN convolution implementation involved unfold->matrix multiplication->fold and was hugely inefficient.
Of course, it would be highly desirable to have a more efficient native implementation, but it is quite a bit of work (e.g. for batch norm I managed to get the wall clock time on the GTX1080Ti close to CuDNN’s but that was a lot easier than I imagine convolutions to be - with things like using FFT sometimes and sometimes not etc.).
Best regards
Thomas