I wish it were so.
I’m using nvml to get the memory readings and there is no match with torch memory functions reports.

First, the cache: If I run torch.cuda.empty_cache(), shouldn’t these 2 returns 0’s?
torch.cuda.memory_cached(), torch.cuda.max_memory_cached()
they don’t. Why?

Calling  empty_cache()  12 can release all unused cached memory from PyTorch so that those can be used by other GPU applications.

“can” == “will”? that’s very ambiguous if it can’t state exactly what it does.
I understand that probably torch.cuda.max_memory_cached() measures the highest cache ever allocated during a given process and it never goes down. It’s of a low utility, but fair enough.
Why doesn’t then torch.cuda.memory_cached() report 0 after the cache was cleared?

Then comes torch.cuda.max_memory_allocated(), which again is of a very little utility, since it appears to be showing the max ever memory allocation, so once you hit some peak, you no longer can measure any peaks below the highest peak.
Let’s compare the torch memory reports with nvml reports in MBs (I call torch.cuda.empty_cache() before each measurement to kill the caching effects - and hoping that it actually does what it advertises to do).
Report columns:
A torch.cuda.memory_allocated() (delta)
B torch.cuda.max_memory_allocated() (delta)
C nvml’s used memory (delta)
D nvml’s peak memory measure via a peak monitor thread (delta)
          A    B    C     D
epoch 1  25 6050    80	6220
epoch 2   0    0     0	 924
epoch 3   0    0     0	 924

Note how column B is stuck at the same number, whereas column D shows that the peak memory used is different between the first epoch and the rest. I double checked that by watching the live nvidia-smi outputs. The GPU RAM consumption peaks at around 7GB during the first run and only at 1.6GB during consequent runs.
What pytorch needs is to have a stop/start method to reset the max measurements, like tracemalloc does. Otherwise, it’s like a depth gauge, where it only indicates the deepest depth one has gone to, but gives zero indication on the actual depth profile.
You can also see from the little table, that the actual GPU memory usage is larger than what pytorch reports. That means that pytorch isn’t reporting the extras the gpu card allocating beyond what pytorch requested. Albeit the difference is not huge.
Also, to clarify, I could easily measure peak memory usage if I were to deal with low-level code. Just take a snapshot of used memory before and after. What I’m trying to profile is top-level code that does internally a whole bunch of various allocations and deallocations.
So at the moment using a peak memory monitoring thread seems to be the only viable solution for measuring intermediate steps. And staying away from pytorch’s memory reporting functions seems to help as well, going directly for nvml reports, getting it directly from horse’s mouth, because that really is the only thing that counts when it comes to OOM problems.
Please don’t get me wrong, I’m not saying that pytorch’s memory reporting functions are incorrect. I’m just saying that they show only part of the picture and more detailed views are needed to be able to tune up one’s code more efficiently.
We are talking about a single process using single GPU setup here, so there is no interference from other processes.
I just hope that torch.cuda.empty_cache really works by releasing all of its cache. Since if it doesn’t then the measurements are incorrect.
edit: I changed the numbers in the table to deltas (before and after), so it’s easier to make sense out of it.