TypeError: __init__() got an unexpected keyword argument 'forward_prefetch'
### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@AlexWertheim 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. run stanford-alpaca's training command: https://github.com/tatsu-lab/stanford_alpaca
```
torchrun --nproc_per_node=4 --master_port=<your_random_port> train.py \
    --model_name_or_path <your_path_to_hf_converted_llama_ckpt_and_tokenizer> \
    --data_path ./alpaca_data.json \
    --bf16 True \
    --output_dir <your_output_dir> \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LLaMADecoderLayer' \
    --tf32 True
```

### Expected behavior

```
Traceback (most recent call last):
  File "train.py", line 231, in <module>
    train()
  File "train.py", line 225, in train
    trainer.train()
  File "/home/projects/transformers/src/transformers/trainer.py", line 1644, in train
    return inner_training_loop(
  File "/home/projects/transformers/src/transformers/trainer.py", line 1731, in _inner_training_loop
    model = self._wrap_model(self.model_wrapped)
  File "/home/projects/transformers/src/transformers/trainer.py", line 1469, in _wrap_model
    self.model = model = FSDP(
TypeError: __init__() got an unexpected keyword argument 'forward_prefetch'
```
The error is raised at the trainer.py:
```
                if type(model) != FSDP:
                    # XXX: Breaking the self.model convention but I see no way around it for now.
                    self.model = model = FSDP(
                        model,
                        sharding_strategy=self.fsdp,
                        cpu_offload=cpu_offload,
                        auto_wrap_policy=auto_wrap_policy,
                        mixed_precision=mixed_precision_policy,
                        device_id=self.args.device,
                        backward_prefetch=self.backward_prefetch,
                        forward_prefetch=self.forword_prefetch,
                        limit_all_gathers=self.limit_all_gathers,
                    )
```
I think forward_prefetch is not supported in PyTorch1.12. Is there a possible solution to enable me to use FSDP with PyTorch 1.12? If not, I suggest adding some version-checking codes.
FSDP support in Transformers requires PyTorch 1.12, so no. You should have hit [this error](https://github.com/huggingface/transformers/blob/55dae94c0ccd088003aa46bcecb2e55321a7f00b/src/transformers/trainer.py#L429) before anything else, not sure why you did not.<|||||>Hi, thanks for your reply. This is not an issue with FSDP support. It's an issue that FSDP does not support the keyword argument "forward_prefetch" in torch1.12<|||||>Hi, I met the same problem with transformers==4.27.1 and the solution is to degrade to transformers==4.26.1 .This may be a version compatibility issues for hugging face transformers .<|||||>Oh thanks for clarifying. cc @pacman100 