Hi
I am new to pytorch
I want to ask about the most efficient way of doing validation step. My training data is about 30 mil, batch size is 128 and doing validation step with about 15% of the data every 2000th training step, to check for early stopping. Training happens on gpu and currently validation step as well. (I have to stick to this 15% fraction, cant reduce it for an external reason)
I notice that validation step is taking a lot of time, almost as much as the 2000 training step. 2000 training steps take about 250 s while validation takes about 160 s. Is this typical? This looks quite inefficient. I tried increasing the validation batch size but i run out of cuda memory. Should i do it on cpu instead? Is there a general wisdom about this that i dont know about?
Thank you for your answers.