Solved by acgtyrant in post #4 
              
              
                What I want is below: 
if CONFIG[config_key].RPN_PRE_NMS_TOP_N > 0:
    # TODO: omit 0 dim bug for PyTorch
    _, order = probs.sort(1, descending=True)
    keep = order[:, :CONFIG[config_key].RPN_PRE_NMS_TOP_N]
    index = torch.arange(0, boxes.size()[0]).unsqueeze(1).long().cuda()
    boxes = boxe…