You are detaching some activations in the forward method in:
  rpn_score = rpn_score.detach().requires_grad_(True)

  rpn_score = rpn_score.to(device)

  rpn_loc = rpn_loc.detach().requires_grad_(True)

which will (as the name suggests  ) detach these tensors from the computation graph and no gradients will be calculated for all previously used layers.