Add GPTBigCode model (Optimized GPT2 with MQA from Santacoder & BigCode)
The GPTBigcode model from BigCode. It is the same model as GPT2, with:
* Added support for Multi-Query Attention (https://arxiv.org/abs/1911.02150)
* A large number of optimizations, mostly targeting inference but also useful in training.

Other than MQA, it's the same model as GPT2, just a new implementation (though it's not numerically equivalent and the checkpoints are not compatible)

The optimizations (I might be missing some):
* Use `gelu_pytorch_tanh` (see #21344 #21345)
* Avoid unnecessary synchronizations (added to GPT2 in #20061, but wasn't in the original santacoder).
* Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).
* Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`
* Rename `scale_attn_by_inverse_layer_idx`-> `scale_attention_softmax_in_fp32` and change its behavior to match Megatron-LM (divide by layer_idx in fp16, then multiply in fp32).
* Cache the attention mask value to avoid recreating it every time.
* Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.
* Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.
* Merge the key and value caches into one (this changes the format of `layer_past`/ `present`, does it risk creating problems?)
* Use the memory layout (self.num_heads, 3, self.head_dim) instead of (3, self.num_heads, self.head_dim) for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible).

Excluded from this PR (optional/opt-in features, could be added later):
* CPU optimization for inference, aka InferenceRunner (huge speedup for generation with pre-allocated tensors, pre-computed views and support; faster than Deepspeed, but too experimental to add now)
* KV cache pre-allocation and padding. (Same reason)
* MQA with separate Q and KV (MQA2 in bigcode, a bit faster for training , slower for inference)
* FlashAttention (planning to add support in near future)
* Conversion script for Megatron weights (the MQA part needs the BigCode fork of Megatron)


TODO:
* Update/fix the tests
* Update the docs (should be mostly ok by now)
* Address the remaining circleci issues (mostly related to the tests)
@lvwerra @harm-devries
(Replaces #21253)<|||||>Code on the Hub is fine too and we are adding better support for it every day :-)<|||||>Hi @sgugger, the next generation of the model will also support this architecture so there should also be significantly more usage. Discussed this also with @LysandreJik previously, what do you think?<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>If you prefer @lvwerra and if the architecture is frozen: we won't be able to accommodate changes after it's merged and released in Transformers (no breaking changes in Transformers), whereas it's easier to quickly experiment with code on the Hub. If you feel the model is mature enough and it's time, I'm not opposed :-)<|||||>Thanks a lot for your feedback! Just addressed them all, 
Small note that the cpu/disk offload seem to not work on the testing suite, but I think it is related to the corner case issues we faced with tiny T5 models, as the test pass for the `GPTBigCodModelTest` but does not pass for the `GPTBigCodeMQAModelTest`.
I will also make sure doctests pass before merging<|||||>Please wait a bit before merging, I'll do a final check for the latest changes<|||||>I did a few minor tweaks, I'm OK for merging if it works for everyone. (Assuming CI passes)<|||||>any updates on supporting flash attention ? or do we have a different PR to track it<|||||>cc @younesbelkada I think this is supported in [BetterTransformers](https://huggingface.co/docs/optimum/bettertransformer/tutorials/convert) no? <|||||>Indeed this should go into `BetterTransformer` API on optimum library: https://github.com/huggingface/optimum 
Once the feature is added there, you can just call `model.to_bettertransformer()` and benefit from flash-attention backend. @bharadwajymg would you mind opening a ticket there and request for BetterTransformer support for GPTBigCode model ? thanks!