Have you had any progress on this?
I too am in a similar situation with FasterRCNN… except I am using ResNet 101… my prior implementation I could fill the GPU with batch_size = 1 … where as with this torchvision version I cannot seem to get above 40% VRAM utilization… which to me indicates down sampling
Is the Integer max_size input of class FasterRCNN(GeneralizedRCNN): changing anything for you in terms of VRAM usage ?  I see no change no matter what number I place in there.
my images are 4K, 3840 x 2160, and up… so they definitely would have trouble before fitting on my GPU which has only 6GBs of VRAM… now I am puzzled how I can maximize the resolution and avoid at all costs down sampling