ttbb:

If I’m right, groups can also be used to split the load on many GPUs, if groups=2, split on 2 GPUs (original AlexNet was trained that way on many less powerfull GPUs)


You could achieve the same output by splitting the input to two GPUs, calling each conv individually, and concatenating the output, as with groups=2.
However, groups=2 will not split the conv to two GPUs. Instead it will use the explained method as if you are using two conv layers side by side. The operation will still use a single GPU.