After updating to 1.8.1 I’m not getting the radix_sort error anymore, but I’m getting the same error as @MCvin. I’m using the stable version with CUDA 11.1 on Ubuntu 18.04. Everything works when run on CPU.
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/pl_bolts/models/detection/faster_rcnn/faster_rcnn_module.py", line 112, in training_step
    loss_dict = self.model(images, targets)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/generalized_rcnn.py", line 97, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/rpn.py", line 364, in forward
    loss_objectness, loss_rpn_box_reg = self.compute_loss(
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/rpn.py", line 296, in compute_loss
    sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/_utils.py", line 46, in __call__
    positive = torch.where(matched_idxs_per_image >= 1)[0]
RuntimeError: CUDA error: device-side assert triggered

The error follows a long series of messages like:
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

I guess the problem is with the torch.where operation in torchvision/models/detection/_utils.py. I’ve been able to run my program moving the variable matched_idxs_per_image to CPU in the __call__ function of class BalancedPositiveNegativeSampler, like:
for matched_idxs_per_image in matched_idxs:
    matched_idxs_per_image = matched_idxs_per_image.cpu()
    positive = torch.where(matched_idxs_per_image >= 1)[0]
    negative = torch.where(matched_idxs_per_image == 0)[0]

from line 44.
Hope that helps.