I’m terribly sorry for not noticing this message earlier.
The largest batch size is 1.
But 1 batch contains 5 images which are of a certain resolution say [1920, 1080]
So the shape of one batch is [1, 5, 3, 1920, 1080].
Downsampling the image in this situation is not possible due to objective, and the per forward pass all 5 images are required. I think the only way to address this problem seems to be to increase the GPU VRAM, but I am already tried using a Tesla V100.
I was wondering, if considering there are two GPUs with say 16 GB + 16 GB, is there a way to distribute inference across both of them (via ModelParallel) or otherwise, to potentially process them this way?
And is there any other way to deal with this? I was interested if this was possible.