@Milas There was a bug in DDP that didn’t take into account evaluation mode, but this was only introduced in https://github.com/pytorch/pytorch/pull/18953 4 which was merged 2 weeks ago. Not 25 days ago when you first started noticing this issue. This issue was fixed last night with https://github.com/pytorch/pytorch/pull/19897 4 and https://github.com/pytorch/pytorch/pull/19901 4.
Can you confirm this this still an issue with the current nightly build?