@mrshenli
I also observed a similar result.
This is the thread I created in the discussion.

  
    
    
    Pytorch Model Parallel Best Practices: Pipeline Stats distributed
  
  
    I am trying to replicate the model parallel best practices tutorial. 
Model Parallel Pytorch Docs 
I use Tesla K80 GPUs for running the example. I didn’t plot graphs but I have the following stats. 
Single Node Time:  2.1659805027768018 
Model Parallel Time:  2.23040875303559 
Pipeline 20 Mean: 3.496733816713095 
I don’t get the best results at this split size and it could be okay, depending on the hardware, software issues this can be possible. So I went for testing what is going on. Then I ran…
  


I also benchmarked this using multiple configurations.
I am not sure about the concurrent run of the code. So I changed it as follows and got a sort of fine result.
class PipelineParallelResNet50(ModelParallelResNet50):
    def __init__(self, split_size=20, *args, **kwargs):
        super(PipelineParallelResNet50, self).__init__(*args, **kwargs)
        self.split_size = split_size

    def taskA(self, s_prev, ret):
        s_prev = self.seq2(s_prev)
        ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

    def taskB(self, s_next):
        s_prev = self.seq1(s_next).to('cuda:1')
        return s_prev

    def forward(self, x):
        splits = iter(x.split(self.split_size, dim=0))
        s_next = next(splits)
        s_prev = self.seq1(s_next).to('cuda:1')
        ret = []

        for s_next in splits:
            # A. s_prev runs on cuda:1
            # self.taskA(s_prev=s_prev, ret=ret)
            with concurrent.futures.ThreadPoolExecutor() as executor:
                futureA = executor.submit(self.taskA, s_prev, ret)
                futureA.result()

            # B. s_next runs on cuda:0, which can run concurrently with A
            with concurrent.futures.ThreadPoolExecutor() as executor:
                futureB = executor.submit(self.taskB, s_next)
                s_prev = futureB.result()

        s_prev = self.seq2(s_prev)
        ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

        return torch.cat(ret)