Ok, do you mean that you want no bias to be applied for input patches that look like
1 0 1
0 0 0
1 0 1

? I think that is trickier to achieve since it almost looks like some kind of data-dependent control flow. However, I would be surprised if this made a large difference in the accuracy of a model though, as these are exactly the kind of patterns that I would expect a convolutional model to learn around/adapt to.