Ruokai:

Here the filter size is (batch_size, number of filters, H, W).


The filter shape does not contain the batch size, as layers are created independently from their batch size. nn.Conv2d filters use [out_channels, in_channels, height, width] instead.
If I understand your use case correctly, you would like to use a grouped convolution as seen here:
x = torch.randn(1, 3, 4, 4)
conv = nn.Conv2d(3, 6, 4, groups=3)
out = conv(x)
print(out.shape)
> torch.Size([1, 6, 1, 1])