I’ve been working on tools for memory usage diagnostics and management (ipyexperiments  14) to help to get more out of the limited GPU RAM. The features include tracking real used and peaked used memory (GPU and general RAM). The peak memory usage is crucial for being able to fit into the available RAM.
Initially, I was spinning off a thread that recorded peak memory usage while the normal process runs.
Then I discovered that I can use python’s tracemalloc 37 to measure the allocated general RAM, but more importantly peak memory usage:
import tracemalloc
tracemalloc.start()
do_someting_that_consumes_ram_and releases_some()
# show how much RAM the above code allocated and the peak usage
current, peak =  tracemalloc.get_traced_memory()
print(f"{current:0.2f}, {peak:0.2f}")
tracemalloc.stop()

So I no longer need the monitor thread for tracking general RAM but still need it for GPU RAM. Perhaps there is a way to have a similar to python’s tracemalloc functionality in pytorch? If not, and it’s something feasible, perhaps this can be a feature request? I’m not asking for all of tracemalloc’s features, but just the limited capacity to measure allocated/free gpu ram, as demonstrated in the example above.
Thank you.