You might be accidentally storing tensor, which are still attached to the computation graph, which should be visible by an increasing usage of the device memory.
Your current code snippet looks fine, so you might have the problematic line of code in another function.
Wrap your code into a with torch.no_grad() block during the validation to avoid storing intermediate tensors and the computation graph.