I’m encountering the same issue myself. I believe you also need to modify the construction of roi_heads, as it likely is expecting to recommend boxes on more scales than your RPN generator is expecting. But I haven’t entirely figured out what I’m doing wrong yet. I’ll post if I discover the error