Based on previous experience - continuing the ZeRO solution discussion here will derail your attempt to find core solutions. I highly recommend removing these follow ups and opening a separate thread on HF forums (and tag @stas) and we can continue there.
But the quick answer is that based on your reply - you’re not using deepspeed, you’re just using its launcher. You need to:

create a configuration file which sets up its ZeRO magic.See: DeepSpeed Configuration JSON - DeepSpeed 19

activate deepspeed Getting Started - DeepSpeed 5

use its wrapped model in training Getting Started - DeepSpeed 4

launch it as Getting Started - DeepSpeed 5


That’s when you will see huge improvements.