drop out does not zero gradients, it zeros tensors in feed forward. Zeroing gradients is not good idea as it has been aggregated over all batches so far and may lead to underfitting. optimizer.zero_grad() is used to zero gradients after each forward and backward propagations.
If you do not use torch.no_grad(), gradients in pretrained model will be updated and the result will be overfitted on your test set.