It might be better if you can reduce the issue to a smaller code sample before sharing it?

Add around your inference loop a print for memory usage change (using something like https://stackoverflow.com/questions/938733/total-memory-used-by-python-process 16 for example).
Make a copy of your code out of your main repo
Replace your actual dataset with a bunch of input = torch.rand(some_hardcoded_size) inside the evaluation loop
Remove your instrumentation code (evaluating the accuracy / plotting etc)
Replace your model with a randomly initialized version (don’t load the state dict)
Replace your model with a single Linear layer that perform the mapping you want (with some views maybe)
Change the input/output sizes so that tests run quickly

At each step when you remove something, make sure that the memory behavior is still bad.
Let me know how far you can get. I may be able to look at your code but if it is big, it might take some time to reproduce / pinpoint the problem with the above process as I won’t be familiar with the code.