I got an answer in github (R-CNN predictions change with different batch sizes · Issue #4257 · pytorch/vision · GitHub 6):

This is expected and is due to padding the differently-sized input images with zeros so that all images have the same size (for batching).
If you crop all images to have the same size before feeding them to the model, the batch size shouldn’t influence the predictions.