This looks interesting!
I think firs you need to limit the debugging scope. For example, if you change the microbatch size to 1, will it still cause memory leak? If so, then the issue at least is not microbatching in pipeline parallelism.
On the other hand, have you tried PyTorch native pipeline parallelism API here?