I was doing inference for a instance segmentation model. I found the GPU memory occupation fluctuate quite much. I use both nvidia-smi and the four functions to watch the memory occupation: torch.cuda.memory_allocated, torch.cuda.max_memory_allocated, torch.cuda.memory_reserved, torch.cuda.max_memory_reserved. But I have no idea about the minimum memory the model needs.
If I only run the model in my GPU, then the memory usage is like:

image1338×649 309 KB

10GB memory is occupied.
If I run another training program first to use some memory, and then run the segmentation model’s inference, then the memory usage is like:

image1368×608 288 KB

5GB memory is occupied for the segmentation model.
If I run another training program with larger batch size to use some more memory first, and then run the segmentation model’s inference, then the memory usage is like:

image923×569 170 KB

With 2437MB memory, the model can also run.
So I am confused about the memory usage. Even with the torch.cuda.memory...  like function, I can’t know how much memory my model needs. It’s always changing. It’s hard for me to choose a suitable memory GPU to deploy my model.