The values of filter are set by myself, not a learnable kernel. For only one filter, I tried with:
a = torch.randn(1, 100, 20, 30) # Batch x Channels x Height x Width
kernel = kernel.view(1,1,5,5).repeat(100,1,1,1)
output = F.conv2d(a,kerkel.float(),group=100,padding=2)

and It can work. But I want to apply 10 filters to generate an output of [1, 100, 20, 30, 1, 10], it means that the output is a tensor of 1 batch of 20x30 of 1x10 vector of 100 channels.