Extend Callback API for remote execution of ClearML Experiments
### Feature request

Hi!

Not that long ago, we added support for the ClearML experiment manager as a training callback, and the general feedback has been good so far! However, ClearML is more than an experiment manager alone.

In order to allow users to quickly experiment, ClearML can clone an existing experiment from the UI and then override the originally captured hyperparameters. It does this by injecting the new parameter values into the code at runtime. A user can then schedule and run this edited experiment clone on a remote machine.

But in order for this functionality to work properly, ClearML has to be able to initialize, access and overwrite the training parameters even before they are first used by the Trainer. The current callback implementation does not allow this.

Do you think this is something worth considering to add? I suspect it's rather easy to add a new, very early callback route, but it sounds harder to me to allow a callback to override the training arguments. What do you think?

### Motivation

We received user feedback on our own support slack channel of users trying to run Transformers remotely, but the parameters not being overridden. More advanced ClearML functionality like pipelines and HPO depend on this functionality to properly work.

### Your contribution

We'd be very willing to make a PR, this issue is meant to discuss if you agree that it can be properly added and if so, how you would like to see it practically :) Thank you for the consideration!
Hi there. The `Trainer` is not allowed to modify its own `TrainingArguments`, a design choice we made so that reproducibility and resuming from a checkpoint work properly. This is why the callbacks are also not allowed to change the training arguments. It's probably best for this use case if you simply subclass the `Trainer` API and either add a new method or override what you need.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.