I’m trying to reuse the servers in my university for Data Parallel Training (they’re hadoop nodes, no GPU, but the CPUs and memory are capable). I referred to PyTorch Distributed Overview — PyTorch Tutorials 1.10.0+cu102 documentation which seems to be super high level, can barely get a thing. Should I use DDP/RPC? Any ideas on how/where to get started?
I went through the example in examples/README.md at master · pytorch/examples · GitHub 1 ; still not very clear!
I understand that I should use gloo as the backend, but how do I configure the slave nodes?
I understand that process with rank 0 is the master, but where do I provide the slave IPs? How to get the slave(s) to listen to the master node for new jobs? How do I start the slave processes?
All of them are CentOS 7 servers, 12 cores/64GB per node, got about 12 of these.
Thanks in advance!