Hi @mrshenli,
Thank you very much for the solutions. I have tried the first solution quickly and it can work! I will try it with the Distributed Autograd and Optimizer once I construct the entire network for training. However, one of my concerns is that when we train the network, there are many iterations and epochs, which means we will have lots of forwarding passes of the following:
# On worker 0
emb_lookup_rref = rpc.remote("worker1", EmbeddingTable.forward, args=(input,))
# note that RNN.forward needs to call to_here() on emb_lookup_rref
rnn_result_rref = rpc.remote("worker2", RNN.forward, args=(emb_lookup_rref,))
# similarly Decoder also needs to call to_here() on rnn_result_rref
decoded_rref = rpc.remote("worker1", Decoder.forward, args=(rnn_result_rref,))
final_result = decoded_rref.to_here()

Tons of RRefs (emb_lookup_rref ,  rnn_result_rref , and decoded_rref ) will be created by the rpc.remote . Should I worry about this? Or these RRefs will be deconstructed automatically?
Thank you!!