In order to square-ify images, it is common to crop them, rather than use padding, which could be a problem if images have key information at the edge. An argument for cropping seems to be that you can randomize it for training images, which has the nice side effect of some low-cost data augmentation.
Could you not do the same with padding? I.e. for validation pictures add equal amount of padding left and right (top/bottom for landscape), and for training images randomize the amount of padding on either side?
Anyone know if this is done, and if not, why?