You could implement your own custom convolution using unfold 3 (the docs give an example code snippet for it). To do so you could mask the operation, which would create your desired operation, but wouldn’t yield any speedup. I’m also unsure how you would speedup access and processing of such a “random” pattern.