I’m trying to finetune fastRcnn for object detection on a custom dataset. During each epoch after when I use the optimizer.step() function to update parameters, the model is not working in the evaluation part of the code. However, when I comment the optimizer.step() function I can get the output of the model during the evaluation.  I appreciate any comments.
model = faster_rcnn.fasterrcnn_resnet50_fpn(pretrained=True, min_size=800)
num_classes = 31
in_features = model.roi_heads.box_predictor.cls_score.in_features

#replace the pre trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
epochs = 2
model.to(device)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)


for e in range(epochs):
  model.train()
  loss_history = list()
  for frame, label in training_loader:
      label = [{k: v.to(device) for k, v in t.items()} for t in label]
      frame = frame.to(device)
      losses_dict = model(frame,label)
      losses = sum(loss for loss in losses_dict.values())
      loss_history.append(losses.cpu().detach().numpy().item())
      optimizer.zero_grad()
      losses.backward()
      optimizer.step()
  model.eval()
  for data, label in validation_loader:
      data = data.to(device)
      output = model(data)
      print(output)


When I don’t use the optimizer.step() the print function is:
[{‘boxes’: tensor([[ 814.4761,    4.1513, 1232.4717,  750.0000],
[ 553.6875,  124.6697, 1333.0000,  573.3456],
[ 633.8101,  279.4264, 1333.0000,  661.9217],
[ 619.6268,    0.0000, 1055.0597,  739.1709],
[ 440.1299,  210.7000, 1203.8578,  750.0000],
[ 421.0741,  663.9869, 1100.3203,  750.0000],
[ 604.5875,    1.8037, 1333.0000,  325.6623],
[ 349.5059,  146.8048,  807.5912,  750.0000],
[ 937.7691,  210.4664, 1333.0000,  750.0000],
[ 745.6071,  387.2783, 1173.9806,  750.0000]]), ‘labels’: tensor([12, 12, 12, 12, 12, 12, 12, 12, 12, 12]), ‘scores’: tensor([0.0570, 0.0562, 0.0555, 0.0540, 0.0523, 0.0520, 0.0512, 0.0502, 0.0501,
0.0501])}]
When I use the optimizer.step() the print function is:
[{‘boxes’: tensor([], size=(0, 4), grad_fn=), ‘labels’: tensor([], dtype=torch.int64), ‘scores’: tensor([], grad_fn=)}]