You might want to take a look at the finetuning object detection tutorial: TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.8.1+cu102 documentation 8 as that provides a walkthrough of the scenario you are describing