Are you sure the memory leak is on the inference stage? If not could you post the training loop.  If you accumulate the losses on a list or something similar without calling .item() on it the gradients will accumulate.