I’m running a resized Unet inference. The max input size it would accept is 1280x1280, for this pytorch caches around 4.5 GB of memory. At some edge cases like 505x960 (not a good aspect ratio)input is given the cache memory used is 7.8 GB. This doesnt happen often.
I’m running 2 workers, GPU has 16 GB of memory. I get OOM for some edge cases as mentioned above.