Sure. The model is the version of Faster RCNN found here vision/torchvision/models/detection at master · pytorch/vision · GitHub, and the code I’m working with is largely based on that and the code found here vision/references/detection at master · pytorch/vision · GitHub ; however, I have made some minor modifications, and the version I’m using likely no longer matches the one on github.
I might also note that I began this project with torch 1.5.0 and torchvision 0.6.0. Since recognizing this problem with reproducibility, I am now trying upgrading to torch 1.8.1 and torchvision 0.9.1 so that I can use the torch.use_deterministic_algorithms(), though if I could solve this problem with my older versions I think that might suit me better since I’ve copied files from torchvision 0.6.0 to my repo.
The model definition is as follows
    '''
    model = torchvision.models.detection.__dict__[args.model](num_classes=num_classes, pretrained=args.pretrained,
                                                              **kwargs)
    '''
    model = mymodels.__dict__[args.model](num_classes=num_classes,
            pretrained=args.pretrained, roi_drop_pct=args.roi_drop_pct, **kwargs)

The commented out line is essentially the same, but I’ve created my own mymodels directory with my updated versions of faster_rcnn.py and related files. At this point, the differences between mymodels and torchvision.models.detection are minimal, though I plan to make larger modifications soon. Some relevant arguments are as follows and show my model is [fasterrcnn_resnet50_fpn]github_link_redacted_new_user_limit/pytorch/vision/blob/59c6731897c2b8c48431136515ee80d235c9c2d1/torchvision/models/detection/faster_rcnn.py#L298), though, again, my version is older than on github.
Namespace(aspect_ratio_group_factor=3, batch_size=12, device='cuda', dist_url='env://', distributed=False, model='fasterrcnn_resnet50_fpn')

Input tensors are images from 1080p video frames cut in half. In training the inputs, images, are as follows
-> loss_dict = model(images, targets)
(Pdb++) len(images)
12
(Pdb++) images[0].shape
torch.Size([3, 540, 1920])

I apologize if there’s too much info: I just don’t want to leave anything out. I’d be glad to provide any additional information as well if it might help. Thanks for your quick reply, and hopefully we can get to the bottom of this.