Add SophiaG.
# What does this PR do?

This is a scratch PR showing how to test Sophia with Transformers. This is no way
production ready, and certainly needs to look at licensing. But, this is helpful if someone needs
to try this right away. I'm re-using **AdamW**'s `beta` values. Plus if you see carefully,
there's an ugly hack where I'm using `eps` as `rho`. 

This is code directly copy-pasta-ed from: @Liuhong99 's  [Sophia](https://github.com/Liuhong99/Sophia);

I am putting it here so people can experiment with it and see how it compares to **AdamW**. If there
is sufficient interest in adding this and it can be licensed, would be happy to work on it here. Anyone is free to take
this and turn it into something of value. Please close this as necessary too.

## Before submitting

This PR does none of the above. It is too early to do this, but if there is sufficient interest would be happy to go through this process.

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:
- text models: @ArthurZucker and @younesbelkada

Common:
- trainer: @sgugger

[Semi-related] also linking to Paper page on HF: https://huggingface.co/papers/2305.14342<|||||>Thank you all for encouraging this. 

As a first cut, I am working with the authors to see if this can be a PyPi package if authors agree. License is MIT, so hopefully we can get this out soon.  [Sophia on PyPi](https://github.com/Liuhong99/Sophia/issues/29)<|||||>As Younes said before, we won't merge this PR but can leave it for anyone who want to try this out: Transformers is a library of models, not optimizers (the optimizers inside the library are actually deprecated). Once there is a package supporting this optimizer we can add support for the Trainer like we did for the `bitsandbytes` optimizers.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.