Thank you. I got it working. Hopefully monitoring how this evolves will help me deduce the problem.
I looked at the FasterRCNN meta-architecture definition in the TF code base but it looks quite complex to use in practice. I also notice the use of batch normalization in the RPN, which is not part of the original FasterRCNN model and is not needed in PyTorch implementations I’ve studied: https://github.com/tensorflow/models/blob/bea8998b1974015a01e5aa0e2d80a1c6623798a7/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py#L508
If I can’t resolve this in the next day or two, I will try to write a minimalistic RPN implementation in both Keras and PyTorch to confirm whether the problem is Keras.