You could be allocating a memory size larger than all remaining memory, hence it still OOMs even when the GPU isn’t full.


Are you sure that your nvidia-smi is run when the script is using most memory? CUDA memory usage can change quite quickly sometimes.