Yes, checkpoints are normally used while training the model. If you’re not training your model then there shouldn’t be any gradients to be computed and hence GPU memory usage would be low. However what would be on the GPU would be your pretrained model, input tensors, intermediate tensors and then output tensors. If you’re only using batch size 1 then there shouldn’t be such problem unless you’ve mistakenly kept all your input data on GPU or your input tensors are unreasonably large, but that sounds improbable.
What you can do is to monitor GPU memory usage after each line of code you think would take up memory. You can use GPUtil library:
import GPUtil
.
.
.
out = model(input)
GPUtil.showUtilization()
.
.
.