I found a solution to this issue that works. Please note that this is not a device issue or shape issue, but a deeper issue with the AnchorGenerator used by both Faster R-CNN and Mask R-CNN when using parallelism.
Full issue here: https://github.com/pytorch/vision/issues/1659 164
This is fixed by adding the following line to /site/package/path/torchvision/models/detection/rpn.py on line 409:
anchors = [a.to(pred_bbox_deltas.device) for a in anchors]

Let me know if this works for you; I’ve been able to successfully train on two RTX2080 at once.