Hi!
When I train mask rcnn with resnext101 backbone, Loss goes to Nan.
My code is made from pytorch tutorial.
backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True, trainable_layers=5)
model = MaskRCNN(backbone, num_classes)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

error message :
Epoch: [0]  [   0/1388]  eta: 0:23:07  lr: 0.000010  loss: 2.0952 (2.0952)  loss_classifier: 0.6831 (0.6831)  loss_box_reg: 0.0190 (0.0190)  loss_mask: 0.6889 (0.6889)  loss_objectness: 0.6918 (0.6918)  loss_rpn_box_reg: 0.0123 (0.0123)  time: 0.9999  data: 0.3882  max mem: 0
Loss is nan, stopping training
{'loss_classifier': tensor(nan, device='cuda:1', grad_fn=<NllLossBackward>), 'loss_box_reg': tensor(nan, device='cuda:1', grad_fn=<DivBackward0>), 'loss_mask': tensor(nan, device='cuda:1', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_objectness': tensor(0.6931, device='cuda:1', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_rpn_box_reg': tensor(0.0050, device='cuda:1', grad_fn=<DivBackward0>)}

When I change the backbone to resnet50 or resnet101, no error occurs.
I try to reduce lr but error still occurs with resnext-like.
How can I solve it?