This section gives brief ideas on how to make training faster and support bigger models. Later sections will expand, demonstrate and elucidate each of these.
Faster Training
Hardware:
fast connectivity between GPUs
intra-node: NVLink
inter-node: Infiniband / Intel OPA
Software:
Data Parallel / Distributed Data Parallel
fp16 (autocast caching)
Bigger Models
Hardware:
bigger GPUs
more GPUs
more CPU and NVMe (offloaded to by DeepSpeed-Infinity)
Software:
Model Scalability (ZeRO and 3D Parallelism)
Low-memory Optimizers
fp16/bf16 (smaller data/faster throughput)
tf32 (faster throughput)
Gradient accumulation
Gradient checkpointing
Sparsity
Hardware
Power and Cooling
If you bought an expensive high end GPU make sure you give it the correct power and sufficient cooling.
Power:
Some high end consumer GPU cards have 2 and sometimes 3 PCI-E 8-Pin power sockets. Make sure you have as many independent 12V PCI-E 8-Pin cables plugged into the card as there are sockets. Do not use the 2 splits at one end of the same cable (also known as pigtail cable). That is if you have 2 sockets on the GPU, you want 2 PCI-E 8-Pin cables going from your PSU to the card and not one that has 2 PCI-E 8-Pin connectors at the end! You won't get the full performance out of your card otherwise.
