Hi @fmassa, I think I found the reason: rpn.anchor_generator._cache keeps the pair with the model and the number of tensors it holds expands as proposal grows. I provide a sample 13 for an easy repeat.
BTW, I send a pull request 15 with the fix along with other data type error solution but codecov failed me. Would you please have a look?