@pritamdamania87 Yep, I just modified this official tutorial 7 with DDP.
The code is here:

  
      gist.github.com
  
  
    https://gist.github.com/Euruson/d9bae6490e8b101e0ff9b37c4af1bc76 48
ddp_val.py
# https://discuss.pytorch.org/t/distributeddataparallel-barrier-doesnt-work-as-expected-during-evaluation/99867/3

from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
import torchvision
This file has been truncated. show original




  
  
    
    
  
  


The result is like this:
Rank:0 - Epoch 0/24
Rank:1 - Epoch 0/24
----------
Rank: 0 - train Loss: 0.5293 Acc: 0.7459
Rank: 1 - train Loss: 0.4891 Acc: 0.7623

Rank:1 - Epoch 1/24
Rank: 0 - val Loss: 0.2841 Acc: 0.8889

Rank:0 - Epoch 1/24
----------

It seems that dist.barrier() doesn’t work as rank 1 just goes to the next epoch without waiting rank 0’s validating. And then the program just freezes