OOM error when training with trainer
### System Info

transformers: 4.19.2

When I use trainer to do mlm training on deberta-v3-large, there is out of memory problem.
And the GPU occupancy continues to grow over time, eventually out of memory.and throws OOM error every time at fixed position

By searching forums and issues, I tried to modify the source code of trainer, including

1. in the huggingface transformers trainer code (function create_optimizer), add force_broadcast_object=True
2. rewrite the Trainer saving function to skip saving the optimizer weights
3. disable the optimizer saving by commenting out consolidate_state_dict as well as the optimizer saving part
4. Remove useless intermediate variables and empty_cache

but it didn't work


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


training_args = TrainingArguments(
    output_dir=args.output_dir,
    evaluation_strategy="no",
    learning_rate=args.lr,
    weight_decay=0.01,
    save_strategy='steps',
    per_device_train_batch_size=args.batch_size,
    num_train_epochs=args.num_train_epochs,
    # report_to="wandb",
    run_name=f'output-mlm-{args.exp_num}',
    # logging_dir='./logs',
    lr_scheduler_type='cosine',
    warmup_ratio=0.2,
    fp16=True,
    logging_steps=500,
    gradient_accumulation_steps=args.gradient_accumulation_steps,
    save_steps=5000, 
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    # eval_dataset=tokenized_datasets['valid'],
    data_collator=data_collator,
    # optimizers=(optimizer, scheduler)
)


### Expected behavior

Hope it works without changing batch_size
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.