I’m having the same issue with pytorch 1.8.1 and cuda 11.1.
The error is different though:
~/anaconda3/envs/pytorch-1.8.1/lib/python3.8/site-packages/torchvision/models/detection/_utils.py in __call__(self, matched_idxs)
     43         neg_idx = []
     44         for matched_idxs_per_image in matched_idxs:
---> 45             positive = torch.where(matched_idxs_per_image >= 1)[0]
     46             negative = torch.where(matched_idxs_per_image == 0)[0]
     47 

RuntimeError: CUDA error: device-side assert triggered

I works with:

pytorch 1.8.1 + cuda 10.2
pytorch 1.7.1 + cuda 11.0

Seems like cuda 11.1 is the problem here.
Sorry I couldn’t help more.