Would this help, for ResNet50?
anchor_sizes = ((32,), (64,), (128,), (256,), (512,)) 
aspect_ratios = ((0.5, 0.75, 1.0),) * len(anchor_sizes) 
anchor_generator = AnchorGenerator(anchor_sizes, aspect_ratios)
# get the number of input features 
in_features = modelr.roi_heads.box_predictor.cls_score.in_features
# define a new head for the detector with required number of classes
modelr.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) 
modelr.rpn.anchor_generator = anchor_generator #adding in customized anchors