This sounds like a distributed training setup. While the usual workflow would be to let each node compute a single forward and backward pass on its own, I think it should be possible to apply model sharding in this setup as well. Unfortunately, I don’t have an examplecode snippet ready for it and would recommend to look into the functional API of the distributed package.