As you saying, I checked the outputs of layers, but I didn’t find any invalid values. Instead, I found loss is very large value on first epoch, first batch:
{'loss_classifier': tensor(2.4077e+09, grad_fn=<NllLossBackward>),
 'loss_box_reg': tensor(2.9896e+09, grad_fn=<DivBackward0>),
 'loss_mask': tensor(8.4899e+10, grad_fn=<BinaryCrossEntropyWithLogitsBackward>),
 'loss_objectness': tensor(2.1843e+09, grad_fn=<BinaryCrossEntropyWithLogitsBackward>),
 'loss_rpn_box_reg': tensor(1.9489e+08, grad_fn=<DivBackward0>)}

After update, in second batch, losses go to Nan.
It can be clue of solving the problem?
p.s. I use SGD optimizer, lr = 0.0005 with warm-up starting.