Have you checked the masked conv implement in mmdetection:

  
      
      GitHub
  
  
    

open-mmlab/mmdetection 291

Open MMLab Detection Toolbox and Benchmark. Contribute to open-mmlab/mmdetection development by creating an account on GitHub.


  
  
    
    
  
  


The above masked_conv can work, but the speed is very slow compared with normal nn.Conv2d.