Solved by sudomaze in post #2 
              
              
                After reading the FPN paper [1] again, I have realized that I have something mixed up. Let me share my conclusion, it seems that the MultiScaleRoIAlign is doing is mapping RoIs to which feature level then, in my case, Faster R-CNN will use this feature pyramid as an image pyramid to compute head. 
I…