Adding GPTNeoX (Tensorflow version)
### Model description

Huggingface has GPTNeoX model by ElutherAI. It's a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license.

However Huggingface currently has only PyTorch implementation of the model.
I would like to contribute it's corresponding TensorFlow implementation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_
cc: @sgugger @NielsRogge <|||||>How would you run this model in Tensorflow though? I don't think it offers the same flexibility as PyTorch wrt to distributing layers on several devices.

cc @Rocketknight1 <|||||>I believe a TF implementation with distribution of layers across devices is possible using [DTensor](https://www.tensorflow.org/guide/dtensor_overview). We've been in communication with the TF team about DTensor, but we haven't implemented a TF model with it yet. This could be a good model to try it with!<|||||>Note that implementing our first DTensor model will probably be quite challenging @shivance - we'll support you if you try it, but you should expect that the PR will need changes to some of our tests or internal functions to support DTensor, and there'll be several rounds of iteration before it's ready.<|||||>@sgugger @Rocketknight1 Check this out
https://github.com/huggingface/transformers/blob/17a55534f5e5df10ac4804d4270bf6b8cc24998d/src/transformers/models/esm/modeling_tf_esm.py#L102<|||||> I've seen the use of dtensor at KerasNLP deeply and I think I can start working on it if you and @sgugger give me an initial roadmap!<|||||>Hi @shivance, the first task would be to make a port of GPT-NeoX to TF, and then we can start adding `DTensor` layout code to it. You can look at this [ongoing LLAMA PR](https://github.com/huggingface/transformers/pull/24375) to see what you need to do. The biggest thing you have to do is to make a conversion of the existing `modeling_gpt_neox.py` file to `modeling_tf_gpt_neox.py`, then import the classes from it and run some tests to see that it gives (approximately) the same outputs. Once that's done, we can talk about next steps!<|||||>Hey @Rocketknight1 !
I've worked on same as my Google Summer of Code project, check Tf version of GPT Neo X [here](https://github.com/keras-team/keras-nlp/pull/1056), I think what's needed is to do it in Huggingface Design.