How to train large models on many GPUs? (2021)
Why isn't there a framework that does all this automatically for you?<p>I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU.<p>I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over how many GPUs manually.<p>I would be interested if anyone knows of a framework that manages this automatically and just works.<p>EDIT: BTW I am talking about model sharding not data parallelism which works very well with DDP.