I had this problem too when I tried to perform inference on a collection models, say model_1, … model_n.
I solved this by detaching output from the computation graph
output = model_i(input).detach()

I am not an expert in how GPU works. But I think GPU saves the gradients of the model’s parameters
after it performs inference. That can be a significant amount of memory if your model has a lot parameters. You can tell GPU not save the gradients by detaching the output from the graph.