smth:

That’s weird. Since it’s only for predictions, are they run in a with torch.no_grad(): block to hold no temporary buffers?


no_grad, yes, but there was some extras setup code consuming GPU, I will try to break it down into functional pieces to understand it better, but the bottom line is the same.



 smth:

at the Python level, yes. Using the garbage collector’s inspector.
How to debug causes of GPU memory leaks? for one code snippet to do this.


Perfect. Thank you for that link! I am going to experiment with that code next.
Thank you for your help, @smth.