Solved by tom in post #7 
              
              
                NMS is one of those operations where it is common to write custom kernels because, as you note, implementing this in PyTorch directly is not so fast. 
In the MaskRCNN-Benchmark-Implementation there is are CPU and GPU kernels for NMS, e.g.: 
 
 
Best regards 
Thomas