If you haven’t wrapped the validation step into a with torch.no_grad() guard to save memory, you should do it. 
Besides that there is no secret and the timing might be expected depending on your model, data loading etc.