Optimizer being initialized twice when using Nvidia APEX mixed precision backend
### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.4.0-128-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
- GPU : Nvidia A4000

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine tune models on Jupyter notebooks. When training for the first time using APEX by calling `trainer.train()` it works fine. But if I start another run or try to train another model without restarting the kernel and calling the same function I get an error `RuntimeError: A given optimizer should only be passed through amp.initialize once`  . I thought it was because I had used the `adamw_apex_fused` setting for the optim in the`TrainingArguments`, but it happens regardless of whether or not I specify a parameter for the optim. I believe this is a possible bug due to `transformers` calling the `amp.initialize` function every time the trainer is started and does not check if `apex.amp` has been instantiated. An example of my training code is provided below, I would greatly appreciate any help. Thanks!

```python

training_args = TrainingArguments(
    f"gpt2-wikitext",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    half_precision_backend="apex",
    tf32=True,
    fp16=True,
    auto_find_batch_size=True,
    optim='adamw_apex_fused'
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["validation"],
)

trainer.train()

```


### Expected behavior

Training should proceed normally without errors if `trainer.train()` is called multiple times in the same session.
I can confirm this is done at every call to train, though I don't see an easy way of undoing it and we stopped actively maintaining the integration with Apex now that mixed precision is upstreamed in PyTorch.<|||||>Ah I see, so I should consider Apex support to be deprecated in `transformers` and move all future code to use native PyTorch MP backend?<|||||>Yes, you'll get better support this way!<|||||>That's great, thanks a lot for clearing things up!