Thanks for your response.
I have understand the 23456.
However,
if I use the DistributedDataParallel function to utilize the multi-GPUs on one server machine,
the init_method=‘tcp;//server_ip:port’ ?
PS:
If I use the DataParallel function,
warning message will be raise while using rnn or lstm.
UserWarning: RNN module weights are not part of single contiguous chunk of memory.
This means they need to be compacted at every call, possibly greatly increasing memory usage.
To compact weights again call flatten_parameters().