Do you reload the model etc. in the val_loader loop on purpose?
Could you move it in front of the loop and check again, if the memory is increasing?
It might be, you are holding some references to the model or other objects on the GPU in one of the “init methods” like plf.PerceptualXentropy or aa.LInfPGD. Thus this memory might be collected, since PyTorch cannot free it. Could you check that or give some info on the implementation of these methods?