Solved by mrshenli in post #3 
              
              
                Yep, the first chart looks correct to me. 

If so, is there any chances to enhance it? Makes it more like this chart below. 

Do you need single-machine multi-GPU pipeline parallel or multi-machine pipeline parallel? 


If it is within a single machine, it is possible to parallelize backward as well…