After debugging for hours I found this line of code solved my problem:
with torch.no_grad():
    valid()  # function of your validation loop

Strongly suggest this should be put into official tutorials and examples.