Hi,
As per my knowledge with pytorch you can do parallel training on multiple GPUs/CPUs on a single node without any issue but its not matured yet to do multinode training without any issues considering asynchronous data parallelism. if its supported on multinode too please provide me a simple example to test it out.
Thanks.