I see, so I am able to reproduce one of the errors I got (perhaps related) with the snippet below:
import torch
import pickle
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

num_classes = 2
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
model.roi_heads.detections_per_img = 10

in_features = model.roi_heads.box_predictor.cls_score.in_features

model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
hidden_layer = 256

model.roi_heads.mask_predictor = MaskRCNNPredictor(
    in_features_mask, hidden_layer, num_classes
)

device = torch.device("cuda")

with open('batch_sample.pkl', 'rb') as f:
    batch = pickle.load(f)

images, targets = batch

model.to(device)
model.train()
targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

loss_dict = model(images, targets)
loss = sum([loss for loss in loss_dict.values()])
loss.backward()

you can download the sample input data in here 3.
My setup is:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 207...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   57C    P8    15W /  N/A |    682MiB /  7982MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+


(venv) vitor@vitor-Oryx-Pro:~/Projects/ocr$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal


Python 3.8.5

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

 torchvision. '0.8.1'

Does it suffice, @ptrblck?