Thanks for the answer. So, it makes sense to GPU usage to dip during validation for the reasons you specified. In that case, since training and validation are independent, is there a way to force GPU to start the training of next epoch while doing validation of the previous epoch ?