Hi Stefan. Yes I do actually have a lot of tips so far. I am learning as I go here, so some of these tips might be super obvious/useless for you. Hopefully nothing I’m saying is outright wrong, I’m on a learning journey so be kind to me internet (lol) but that being said…
Here are some thoughts I have had:


First of all note that openCV reads files in BGR format by default, which I embarrassingly learned after the fact and mistrained a few networks. Further Tensorflow’s decoding of .jpeg and .png files ( tf.image.decode_jpeg, tf.image.decode_png) both return an array in (height, width) not (width, height) format like I had thought. These pretrained models MUST take input in (3 x H x W) format (RGB).


When using Image resizing algorithms, use openCV to make 100 % sure that the result looks correct by drawing the resized boxes in the resized image. I’ve settled with 600 by 600 images with bilinear interpolation, cubic interpolation in tensorflow was giving me strange problems. I even wrote a de-normalization function out of paranoia that the normalization problem was messing with the inputs to see if it matched the original image after normalization (i.e invert the procedure). I think checking the inputs just cannot be stressed enough, especially if you are serializing the input, as I was with Tensorflows tf reocrds. There are a lot of sneaky problems that can occur along the way, so right before feeding your batches to a model, I would check the inputs manually and view the images in open CV with the boxes.


Second you should definitely use the image normalization described in the Pytorch vision models page if you want to use the image net  pre-trained weights. I also wrote a script to find the mean r,g,b values and standard deviations for different datasets like the SUN RGBD dataset, Pascal 2007, 2012 if you want to train from scratch (i.e. not use Imagenet weights, but you still want to normalize channel info)


For the optimizer, almost all papers I am reading for RPNS (and generally for a lot of machine vision models) use some variant of Stochastic Gradient Descent with momentum, with a learning rate scheduler, where for the first portion of training the learning rate is either 0.001, or 0.003
Something like :
 optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, ResnetRPN.parameters()), lr=0.001, momentum=0.9, weight_decay=0.0005)
After some amount of iteration, they decrease the learning rate to 0.0001 for fine tuning. I think this is pretty important. I’m not sure why using the adam optimizer with default settings was so bad, this could have been due to errors on my end, but I am just going to follow what I am reading in most of the papers I am checking out (e.g. Faster - RCNN, Fast RCNN, DP-FCN, R-FCN, etc.)


During training, its worth examining which loss (regression, objectness) is really spiking the loss. For me, it seems like on Pascal 2007, 2012 the objectness improved pretty rapidly, but the regression was really the main source of the error (even after re-weighting as in Faster-RCNN, i.e. dividing by 256 or the number of region proposals in a batch). Questions you can ask are: which anchors are really causing the problem? I.e. Can a smaller 32 by 32 anchor that is not very well centered on an object anchor to learn to grow to a massive 500 by 500 box that well?


The number of iterations in the original faster-RCNN paper seems a bit low to me, my own models benefitted from more iterations (up to 32 times on the same dataset with the 128 region proposals). This may not be good advice lol.


Testing the RPN after training visually is a good way to inspect what might be the issues (boxes all being the same size, missing very obvious edge/colour cues). I wrote a bunch of fake RPNS that do things like randomly sample box size and x_min,y_min location. Or randomly choose anchor boxes as in Faster R-CNN, sample from a normal distribution of box x_min, y_mins with certain heights and widths. I had a few other ones (making even guess in each quadrant, etc.), but you get the idea.
I compared my RPN to these fake ones, and it did much better thankfully. Clearly since the random RPNS don’t use any image information, if the network can’t vastly outperform them you know something is seriously wrong. I was surprised how even with pretty bad training loss how good the network I trained on Pascal 2007 was (finding really small objects, PERFECT bounding boxes, overall good average IOU for higher ranked boxes). Take the 10 most highly ranked (by objectness) boxes and compute their max IOU with the ground truth boxes, if you consistently get IOUs over 0.7 you are in good territory. I’m thinking of writing an article “What Makes a Good Region Proposal Network” actually. The literature is pretty sparse on this topic, there aren’t really any testing metrics agreed upon for this task that I can see.


As for transfer learning, and which layers to freeze and all that jazz… So far I have just not been able to make transfer learning work that great…, but I have recently fixed a lot of problems, so I will get back to you on this one. My best results to date come from training from scratch on MS COCO 2014, 2015 with all layers at the same time, which is very slow, but worth it. All my layers were initialized uniformly at random.


There is also the option of popping off layers from the Resnet 101 backbone in order to get a larger feature map. The stride is quite large I find if you want to do things like position sensitive poolings, compute masks after, you definitely benefit from a larger feature map in some circumstances.


I have wrote a “dilated” Resnet 101 backbone that for various layer configurations, does not pop off layers, and in this way you can still keep the layer configuration and choose based on the size of the feature maps you want by keeping the stride in bottleneck downsample sequential model to be 1 for the 1 by1 conv2d thats used to downsample the image (before batch normalization); the dilation is then set to 2 (although I’d like to experiment with 3,5,7). This may be trickier when using pre-trained weights however since that kernel was not originally trained with any dilation used. I have yet to test these dilated networks, but I will.


Hopefully this helps.