I’m working on a dataset with images of different aspect ratios and different sizes.
The model is quite large and takes a long time to train so I’m using multiple GPUs.
I’m wondering what is the “correct” or at least “reasonable” way to do that.
Should I just resize each batch to be of the same size? Where should it be done - in my Dataset? in collation?
Should my Dataset return a single image, or already batch them and return tuple(images_batch, result_batch)?
Should I calculate the prediction loss and accuracy and other statistics inside the forward pass and just return it? or should it happen outside after recollating them?
I’m looking for common practice.