Thanks a lot for your expaination! My GPU memory is too small for a large batch size which limits my tests. I guess the main purpose of GPipe is to enable us to train extremely large model not to accelerate the training process.
I noticed that in the your experiments, pipeline methods use different batch size. Isn’t this going to affect the model performance?