I would expect the training time to take a hit, because you’re moving much more data around compared to a direct forward/backward. All of that overhead will come at a performance penalty. If I understand correctly, pipelining with this approach is best suited for allowing extremely large models to train if you have limited memory available.