How to train large models on many GPUs? (2021)
Why isn't there a framework that does all this automatically for you?I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU.I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over how many GPUs manually.I would be interested if anyone knows of a framework that manages this automatically and just works.EDIT: BTW I am talking about model sharding not data parallelism which works very well with DDP.