Did you just solve the problem. I faces the same now.
~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in forward(self, images, features, targets)
    362             labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
    363             regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
--> 364             loss_objectness, loss_rpn_box_reg = self.compute_loss(
    365                 objectness, pred_bbox_deltas, labels, regression_targets)
    366             losses = {

~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in compute_loss(self, objectness, pred_bbox_deltas, labels, regression_targets)
    294         """
    295 
--> 296         sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
    297         sampled_pos_inds = torch.where(torch.cat(sampled_pos_inds, dim=0))[0]
    298         sampled_neg_inds = torch.where(torch.cat(sampled_neg_inds, dim=0))[0]

~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/_utils.py in __call__(self, matched_idxs)
     55             # randomly select positive and negative examples
     56             perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
---> 57             perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]
     58 
     59             pos_idx_per_image = positive[perm1]

RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal

OS: Ubuntu 20.04
GPU: RTX 3080
package: pytorch-1.8.0 with CUDA 11.1