Hi, thank you for your question.
There are some conditions to optimize a model with GPipe:

The model requires a large amount of memory.
The original batch size is not so small. Because we need a micro-batch which is not too small. If a micro-batch is too small, GPU wouldn’t be utilized.
Well balanced. The imbalance between partitions makes GPipe underutilized.

I just published my ResNet-101 performance benchmark 19. If you have the same environment with me, I expect you get the same result. Even you don’t have the same environment, the code will be helpful. Especially, you can check the balance of ResNet-101 what I’ve used, and how I measured the training time.