Great idea, thanks a lot ptrblck! I’m working entirely in C++ though (both for training & inference), so no torch.cuda.memory_summary() right now but by looking at the source code of the Python API I found that I can get what I need by playing around with c10::cuda::CUDACachingAllocator::getDeviceStats and the c10::cuda::CUDACachingAllocator::StatType enum.