While you could theoretically avoid computing the tagged pixels and reduce the runtime, you would have to find fast algorithms, which are able to accept a random mask in the input image.
E.g. you could try to write a custom convolution via unfold and remove the patches, which are tagged as invalid, and recreate the output. Once this is done you could compare the performance against a “standard” convolution via unfold and check the performance gains. Note that convolutions are implemented using optimized algorithms, which would most likely outperform the “masked unfold” approach even when they are using the complete image.