Solved by ptrblck in post #3 
              
              
                Yeah, but not out of the box. 
Note that the FasterRCNN ResNet backbone uses FrozenBatchNorm layers, while you would most likely want to use the trainable batchnorm layers. 
Also, the last linear layer is missing and the output is different. 
However, there might be a hacky way to grab the backbone’…