It seems that it is not possible to use image without annotation for the training: https://github.com/pytorch/vision/issues/1128 75
I’ve tried to pass boxes of size 0 but then ‘loss_rpn_box_reg’ becomes inf.