GPTNeoXAttention does not deal with odd numbers of attention heads
### System Info

transformers 4.29, HEAD
Linux (not relevant, reproducible on e.g. Mac OS)
python 3.10.11 (not relevant, reproducible on e.g. python 3.9.13)

### Who can help?

@ArthurZucker
@younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When the hidden size is not evenly divisible by the number of attention heads, GPTNeoXAttention throws an exception while trying to reshape its state.

Here is a minimal example to reproduce:

```
from transformers import pipeline
p=pipeline(model="Isotonic/gpt_neox_225M",task="text-generation")
p("I like to eat ")
```

The exception is the following:

```
/home/jps/anaconda3/envs/transformers/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:133 in forward                                                              â
RuntimeError: shape '[1, 5, 12, 255]' is invalid for input of size 15360
```

### Expected behavior

What happens here is that the model has 12 attention heads and a hidden size of 1024. Thus, head_size is calculated as 1024 // 12 == 85. Then, 3 * head-size is not 256 but 255, giving the issue here.

I am not quite sure how this is supposed to work. In https://github.com/EleutherAI/gpt-neox, the code checks that the hidden size is evenly divisible by the number of heads. This would not enable the use of this model, but it might give a better error message.

Is there any chance to run such a model?
Hey! Thanks for reporting!Feel free to open a PR that raises and error when the hidden size is not divisible by the number of heads. This indeed should not happen.  (Can probably be checked in the config)