I think I’ve come up with a solution by stacking along the channel dimension and misusing groups:
>>> x1.shape
(1, 3, 5, 5)
>>> x = torch.cat([x1,x2,x3,x4], dim=1)
>>> x.shape
(1, 12, 5, 5)
>>> w = torch.randn(12, 3, 1, 1)
>>> y = F.conv2d(x, w, groups=4)
>>> y.shape
(1, 12, 5, 5)
>>> y.view(4,3,5,5)

I’m not 100% sure this is the right thing to do, so I’d still appreciate any thoughts.