In general this can be kind of tricky to reason about, because reserved memory might not always be fully used (e.g., reserved ahead of time to speed up future allocations) and also because allocations happen in blocks and fragmentation means that reserved memory > allocations.
I think the closest thing you can get to a guarantee on the required memory would be to use set_per_process_memory_fraction: torch.cuda.set_per_process_memory_fraction — PyTorch 1.9.0 documentation 10 and to reduce this amount until the model cannot run to see how much memory it needs. For example, you can just keep reducing the fraction, and use the (fraction * total gpu memory) amount to get a good idea of the requirements under normal conditions.
Finally, after getting this estimate, I would recommend provisioning at least 100-200MiB of headroom because the memory usage of non-model things like the PyTorch/cuBLAS/cuDNN libraries may grow over time.