I got the following OOM error, but I cannot understand the message precisely.

OOM: Ran out of memory with exception: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 2; 39.59 GiB total capacity; 31.01 GiB already allocated; 21.19 MiB free; 31.83 GiB reserved in total by PyTorch)

From the message, seems PT reserved only 31.83GB, given the GPU capacity is 39.59GB, it should have ~8 GB free, but it only has 21.19MB free. Does it mean there are 8GB memory not managed by PT allocator? Could you help me understand where the 8GB memory goes?
Is the reserved memory the total memory that is allocated and cached by the memory allocator?
Also, I got the following memory stat print out, seems align with the error message.

image681×614 16.1 KB