I tried to reproduce this issue using this code snippet:
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True).cuda()
images, boxes = torch.rand(4, 3, 840, 1280).cuda(), torch.rand(4, 11, 4).cuda()

labels = torch.randint(1, 91, (4, 11)).cuda()

images = list(image for image in images)
targets = []
for i in range(len(images)):
    d = {}
    d['boxes'] = boxes[i]
    d['labels'] = labels[i]
    targets.append(d)
output = model(images, targets)
print(output)

which works fine on my machine.
Could you post an executable code snippet using random shapes, which triggers this issue, please?