Can you please explain how I can group such that the sample code works?
A = torch.randn(5, 512, 7, 7) # minibatch, in_channels, iH, iW
B = torch.randn(5, 512, 1, 1) # minibatch, filter_channels, h, w 

# Aim is to run conv2d at batch level
out = F.conv2d(A, B)
# However, It applies all the filters to all the batch samples of A
# Desired output shape = [5, 1, 7, 7]
# Obtained output shape = [5, 5, 7, 7]

Using a groups = 512 or 5 raises RuntimeError due to shape mismatch.