I am getting an OOM:
RuntimeError: CUDA out of memory. Tried to allocate 2.58 GiB (GPU 0; 15.78 GiB total capacity; 8.19 GiB already allocated; 1.67 GiB free; 4.96 GiB cached; 0 bytes inactive)  (malloc at ... cuda/CUDACachingAllocator.cpp:382)
There is 4.96 GiB cached along with 1.67 GiB free, yet the torch couldn’t allocate  2.58 GiB.
I am suspecting that malloc was trying to find 2.58GiB contiguous memory block but it couldnt?
My question is: If I force torch.cuda.empty_cache() once in a while (maybe once per epoch or for N number of steps?), will it improve the contiguous block situation between free and cached pools?
Thanks