Thank you for your kind reply! Yes, I think fragmentation is more plausible to me as turning off fragmentation helps mitigate this OOM problem (by setting PYTORCH_CUDA_ALLOC_CONF).
I’m more curious about the fact that most OOM error I have here is raised by trying to allocate all the free memory and fails. The amount of memory “trying to allocate” varis every time, but is always the same(at least in GB) with free memory.
For most CUDA OOM errors I can find online, “trying to allocate” is bigger than “free”, and they can be tracked down to a specific operation, like creating a large tensor.
That’s the special&interesting point in my case. I want to figure out whether my strange OOM error is caused by some tensor creation or by a buggy allocator inside pytorch.
FYI my pytorch version is 1.10.0, coming from the official nvidia docker image.