elif self.fsdp is not None and self.args.fsdp_config["xla"]:
### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.4.143.bsk.8-amd64-x86_64-with-glibc2.28
- Python version: 3.10.9
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

in trainer if self.args.fsdp_config["xla"] is true，trainer will wrap model layers，is this mean if i want to use fsdp to shard model，I must install torch-xla>2.0?

### Expected behavior

want to know if  i must install torch-xla for fspd trainning
Hi @duanzhenyu001, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

That being said, checking the docs it's possible to see that [XLA supports FDSP](https://huggingface.co/docs/transformers/main/main_classes/trainer#pytorchxla-fully-sharded-data-parallel), but [FDSP can be used separately](https://huggingface.co/docs/transformers/main/main_classes/trainer#pytorch-fully-sharded-data-parallel). If you wish to use XLA with FDSP, it's necessary to install torch-xla >= 2.0.  <|||||>@amyeroberts thanks a lot for your reply. when I use huggingface trainer fsdp mode finetune a 6B model，I got oom even 8 V100-32G gpu used. I didn't find where trainer shard my model when I use fsdp without xla. could you please point it out in the trainer code. thanks very much.   <|||||>@amyeroberts  and here is my TrainingArguments when I finetune glm2 a 6B model.
TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_backend=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=64,
eval_delay=0,
eval_steps=None,
evaluation_strategy=epoch,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=full_shard auto_wrap,
fsdp_config={'fsdp_forward_prefetch': True, 'fsdp_sync_module_states': True, 'fsdp_use_orig_params': True, 'xla': False, 'fsdp_transformer_layer_cls_to_wrap': 'GLMBlock'},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=GLMBlock,
full_determinism=False,
gradient_accumulation_steps=128,
gradient_checkpointing=False,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=debug,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune/runs/Jul14_06-57-19_mlxlab25ta1apm6482c909-20230609063905-cqjk4l-u3z93n-worker,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1,
logging_strategy=steps,
lr_scheduler_type=cosine,
max_grad_norm=None,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=2,
optim=adamw_torch,
optim_args=None,
output_dir=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune,
save_on_each_node=False,
save_safetensors=False,
save_steps=50,
save_strategy=steps,
save_total_limit=1,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=1000,
weight_decay=0.1,
xpu_backend=None,
)<|||||>@amyeroberts and at function "create_accelerator_and_postprocess" , 
        self.accelerator = Accelerator(
            deepspeed_plugin=self.args.deepspeed_plugin,
            gradient_accumulation_steps=self.args.gradient_accumulation_steps,
        )
self.accelerator always be inited, if i don't set os env ACCELERATE_USE_FSDP to be true, "self.is_fsdp_enabled = getattr(self.accelerator.state, "fsdp_plugin", None) is not None"  self.is_fsdp_enabled will always be false,  is this what you realy wanted?  and I'm confused the intention of this function.<|||||>cc @pacman100 who will be able to comment on the OOM memory issues and whether this is expected. 

Please note that recently there was a large update with Trainer, and it now uses accelerate in the background. In the issue information, I see that you're using v4.24. I suggest updating to a more recent release to benefit from this update and any bug resolutions in between. <|||||>> 

I'm now using v4.30.2，but still get oom memory errors when training on 8 V100-32g GPUs. @pacman100  can some one give any suggestions? thanks a lot<|||||>Hello @duanzhenyu001, how are you launching the training script and a minimal training script for deep dive is required<|||||>The following works on 4 A100 80GB GPUs with GPT-j (6B model), although the entire VRAM is occupied on each:

```
cd transformers/examples/pytorch/language-modeling

torchrun --nnodes 1 --nproc-per-node 4 run_clm.py   --model_name_or_path EleutherAI/gpt-j-6b --dataset_name wikitext   --dataset_config_name wikitext-2-raw-v1   --per_device_train_batch_size 1   --per_device_eval_batch_size 1   --do_train   --do_eval   --output_dir /tmp/test-clm   --gradient_accumulation_steps 8 --overwrite_output_dir --fsdp "full_shard auto_wrap" --fsdp_transformer_layer_cls_to_wrap "GPTJBlock" --bf16
```

The reason for OOM at your end is plausible due to large seq lengths such as >=1024. In such cases, gradient/activation checkpointing is recommended with `--gradient_checkpointing`.