Could you use model.eval() for both, the validation and testing phase?
This will make sure to disable dropout as well as use the running stats of batchnorm layers instead of the current batch stats.
If you validation results are bad after using model.eval(), then the batchnorm stats might be off.
PS: torch.no_grad() is still useful for both phases, as it will save memory.