Thank you, I found my problem. The reason is that I put output into output backward function.
And thanks for the suggestion, but rpc is memory costing. It copy all parts of model and is not pure model parallel.