I would’ve thought it would be best to have a new scaled size for each batch when training, which is why I have the resample_scale within the training iteration loop (not the epoch loop). Do you think varying the scaling per epoch would also be reasonably effective for data augmentation?
I also use random rotation (usually +/- 5 deg) and brightness (+/- 20%). I might try adding saturation as well. On some older architectures I had, I added a gaussian noise layer during training, I might add this back in between the encoder and the seg/flow/depth heads.
I’ll also have a look at torch.utils.checkpoint to increase batch size. I am already using AMP to take advantage of the smaller memory footprint with fp16 (even if I don’t get the other benefits since I’m on Pascal). I ordered a personal RTX 3090 which should hopefully come in a few weeks. Although this is all for a final year undergraduate project which is due in two months so getting some good training in by the time the 3090 arrives is really pushing it…