Local Attention for GPT2
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Our model uses local attention in some layers (i.e each position can only see the last k=256 tokens in every other layer). We would like to be able to specify this in the config on the model hub.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Right now we can't integrate the 1.3B and 2.7B EleutherAI GPT models because local attention is not supported in transformers.


@patil-suraj is working on implementing GPT Neo over at https://github.com/huggingface/transformers/pull/10848! The 1.3B and 2.7B should be loadable in that architecture once finalized.