dim1 in a kernel is the number of input channels, while dim0 is the number of filters.
You could use a depthwise convolution as shown here:
x = torch.randn(1, 96, 64, 64)

N=5
kernel = torch.Tensor(np.ones((N, N)).astype(np.float32)) # Example 5x5 Kernel
kernel = kernel.unsqueeze(0).unsqueeze(0)
kernel = kernel.repeat((x.size(1), 1, 1, 1))
out = F.conv2d(input=x, weight=kernel, stride=1, padding=N//2, groups=x.size(1))