If you are performing a train step followed by an ‘if step  % ___ == 0’ condition to evaluate a test batch in that same step, YOU MUST CALL model.train() in the next step BEFORE any model param/gradient is updated, this must include optimizer.zero_grad( ).