No, it’s not possible to do it with the groups option.
The only way I see of doing it would be to hackily wrap SpatialConvolutionMap (from Lua torch) in pytorch, but it doesn’t have GPU kernels implemented, so you’d need to run it on the CPU.
Furthermore, if I remember properly, it is unlikely that we will have optimized GPU kernels for SpatialConvolutionMap.