I’m not sure how I could apply your idea to the Faster R-CNN model.
I tried simply increasing the out_features of the Fast R-CNN layer, and training in some ‘frog’ and ‘chicken’ but that did not preserve detection of the existing 91 classes. The final layers of the model look like this:
  (roi_heads): RoIHeads(
    (box_roi_pool): MultiScaleRoIAlign()
    (box_head): TwoMLPHead(
      (fc6): Linear(in_features=12544, out_features=1024, bias=True)
      (fc7): Linear(in_features=1024, out_features=1024, bias=True)
    )
    (box_predictor): FastRCNNPredictor(
      (cls_score): Linear(in_features=1024, out_features=91, bias=True)
      (bbox_pred): Linear(in_features=1024, out_features=364, bias=True)
    )
  )

If I figure out how to manipulate the final layers to add features, or apply the Net2Net idea from the paper you linked, I’ll be sure to post it.