Model parallelize for LlaMA
The LlaMA model currently does not support model parallel training, when will this feature be added?

'LlamaForCausalLM' object has no attribute 'parallelize'
This API is deprecated on existing models and won't be implemented on new ones. Use `from_pretrained(xxx, device_map="auto") to parallelize your model.<|||||>Hi sgugger, I wonder if  `from_pretrained(xxx, device_map="auto")` can be used to train models?<|||||>Yes it can, as long as it all fits on GPUs (like was the case with `parallelize` before).<|||||>Thanks for your reply! I've tried it and it works well. And I noticed that `device_map="auto"` can use CPU or disk offloading. But when I try to train a large model with offloading, I get the GPU OOM error.
I also tried setting a low `max_ memory` to set aside buffer for optimizer states in GPU, but still get the GPU OOM error.
Does `device_map="auto"` enable training large model with offloading?<|||||>Yes, if you have the hardware for it. To train a model, you need 4x the size in GPU memory (one for the model, one for the gradients and two for the optimizer state if using Adam). If not, then you will need to look at techniques like Zero-3 in DeepSpeed or FSDP, or use `peft` for fine-tuning.<|||||>Thanks. My point is that using `device_map="auto"` can **offload** some modules of large model to CPU or Disk, so do we still need DeepSpeed or FSDP?<|||||>As I said before, CPU and disk offload with `device_map="auto"` do not support training, only inference.<|||||>Will device_map = "auto" support pipeline parallel training?<|||||>> 

Hi，will model paralleize and zero strategy  work together when training? or any plan for that? Thanks!<|||||>No, ZeRo-3 from DeepSpeed has its own way of initalizing the model which is supported in Transformers already. You shouldn't mix it up with `device_map="auto"`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.