I tried with small matrix and custom conv_filter, it works fine.

conv_filter = torch.Tensor([[[[1, 0, 0]], [[0, 1, 0]], [[0, 0, 1]]]])
x = torch.randn(1, 3, 5, 4)
output = F.conv2d(x, conv_filter, padding=0)

Than you for your help