Hi Yi Wang,
Thanks for your reply! When I limit the microbatch size to 1 and change the num_batches to 10, the memory leak problem exists. I will continue to debug this, any suggestions?
Thank you for recommending this API But PyTorch native pipeline parallelism API seems not to work for multiple distributed machines, right? Currently we want to build pipeline parallelism for multiple nodes.
Best,
Yang