again,  my suspicion is that the computation graphs are created during validation routine.
Can you try to avoid it?
Pytorch <= 0.4:
input = Variable(input, requires_grad=False)

Pytorch >= 1.0:
with torch.no_grad():
    ...

@ptrblck: do you have the similar hunch about this?