Small location errors for large objects will have less influence on the IOU than on small objects.
Here 3 is a good explanation on the IOU score.
Additionally, I think small objects might generally be “hard” examples in comparison to large ones.
Maybe Focal loss (RetinaNet) 2 might help in your use case.