Piotr, that was exactly the solution. Thanks for posting, I forgot to write this down after my Eureka moment yesterday 
For anyone else wanting to do something similar, it was really a case of something like this:
class MaskedConvolution(nn.Module):
    # ...
    def forward(self, x: torch.Tensor, mask: torch.BoolTensor) -> torch.Tensor:
        o = self. .... # do fancy operations here
        assert o.shape == mask.shape, "Mask shape should be the same as the output shape"
        return o * mask