Ajinkya Tejankar

Also, I went through TRL code and couldn't find any instances of llm.sleep() or enable_sleep_mode to put vLLM to sleep.

Thanks. Do you anticipate FSDP support becoming more stable? I am not too comfortable with DeepSpeed since it seems a bit opaque, but if FSDP is buggy, then I'd rather deal with DeepSpeed.

Is DeepSpeed planned to be the main engine to support multi-gpu and multi-node setups for TRL in the future? I tried FSDP and it doesn't work with a bunch of configurations that work with DeepSpeed. For instance, I couldn't get GRPO + FSDP + LoRA + VLLM colocate to work, but swapping FSDP with DeepSpeed works. Is DeepSpeed more reliable than plain FSDP from PyTorch?

PS: Great blog! Thanks a lot for your efforts :)