Hi Milas! This looks very odd. Is it possible that you’re seeing some data contamination between your training and validation datasets? The fact that you run with no_grad during evaluation mode, as well as setting model.eval() all sounds perfectly normal.
You say that if you run without DistributedDataParallel you don’t observe this issue? Does this happen for any number of processes > 1?