Thanks, I will try to dig in and see if I can run the RPN portion but my objective is really to replicate this on my own. I have an implementation that does learn but the mAP is low and convergence takes much longer than it should. I suspect numerical instability.
The chief differences in my implementation are:


VGG16 backbone (none of the TF ones seem to use this).


Single class output in the RPN per anchor (objectness score) with sigmoid rather than two (object score, background score) with soft max. Background score is unused by the model anyway.


My truth values come in a more complex form because I pass a single large map stuffed with class and regression ground truth data.


I think it would be helpful to be able to print out the norm of the gradient but is there an example of how to do this with the current Keras API?