Thanks for tip!
Hi I am running in google collab. I managed to make the code to run but now  I have new issue. I used pretrained weight with coco (101-fpn) from the maskrcnn benchmark, however the learning rate decay did not updated.
params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=5,
                                                   gamma=0.1)  

Epoch: [29]  [500/583]  eta: 0:01:00  lr: 0.005000  loss: 0.4555 (0.4845)  loss_classifier: 0.0889 (0.0954)  loss_box_reg: 0.1322 (0.1424)  loss_mask: 0.2333 (0.2222)  loss_objectness: 0.0111 (0.0123)  loss_rpn_box_reg: 0.0102 (0.0124)  time: 0.7206  data: 0.0151  max mem: 7938
Any advice would be of great help