I actually would like the scale jittering to be applied during training as described in Section 3.1. i.e each training image should be isotropically-rescaled with shortest edge S being randomly sampled from the range [256, 512]. And only then cropped to the fixed size of 224.
Indeed RandomResizedCrop brings a variety of scales and aspect ratios into the training data. However, it results in a different distribution of data then random rescaling of the whole image as suggested in the paper.
Giving a  context to the topic I want to reproduce generalization experiments on VOC2007 described in Appendix B. As I use torchvision pretrained weights for vgg16 I am getting 72.06 mean AP which is around 17 points less than the reported score. Thus, I was suspecting the difference might be caused by vgg weights not being trained with scale jittering.