Based on your first code snippet you are adding another fasterrcnn_resnet50_fpn to the initial model.roi_heads.box_predictor, which would expect the same inputs and targets as model.
I guess the preceding layer of model.roi_heads.box_predictor doesn’t output the expectged input and target tensors, so that the “nested” fasterrcnn_resnet50_fpn model will fail.
Is this indeed your use case or were you planning to add another module to model.roi_heads.box_predictor?