I’m currently trying to detect in advance if a particular model + gpu + PyTorch version combination will run into an out-of-memory issue without actually running the model on the device.
torch.cuda.max_memory_allocated() is working well to verify if I can correctly predict the minimum memory requirements of the model tensors on the GPU.
However I was wondering if there was any way to deterministically predict the overhead imposed by PyTorch + CUDA?