Yes, this would be one way to go. You would need to set the unwanted weights to zero and the gradient as well if that’s your use case.
Here is a small code sample:
out_channels = 6
in_channels = 1
kh, kw = 3, 3
weight = torch.randn(out_channels, in_channels, kh, kw, requires_grad=True)
with torch.no_grad():
    weight[:, :, 0, 0] = 0.
    weight[:, :, -1, 0] = 0.
    weight[:, :, 0, -1] = 0.
    weight[:, :, -1, -1] = 0.

x = torch.randn(1, in_channels, 5, 5)
output = F.conv2d(x, weight)
output.mean().backward()
print(weight.grad)