Is it okay to define and use new model by using only a part of full GPT model blocks? Or anyone tried to do so?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

As far as I know, the smallest GPT2 model that comes with pretrained weight has 768 hidden size with 12 transformer blocks.

However, I'm considering to define 'smaller' GPT model with only 1 or 2 transformer blocks and load those weight from corresponding blocks in the pretrained GPT weight.

This is basically because the original model size is too big to fit in my gpu environment.

so I'm wondering if it will work or not. I think it would be at least better than start training from randomly initialized weight, but not sure about it.

So I'd like to ask how you think. 
Thanks!
### Hi, 
As far as I remember from my experiments with GPT2, the GPT2 blocks will work on its own. What I mean with that is that **the model will produce reasonable outputs, even if you leave out some layers from the original model.** So your idea should work. 
What I'm not quite sure about is the performance of an ONLY 1-2 layers model. I would guess the model will produce valid sentences, however the quality might be not optimal. 

### Another Tip: 
If you want to build this model, load the original GPT2 model from the lib and delete the layers you don't want manually. This is a relatively ugly solution but its very time efficient ). Also you dont have to define you own forward function etc.<|||||>Did you take a look at `distilgpt2`? It has 6-layer, 768-hidden, 12-heads, 82M parameters.

cc @VictorSanh @LysandreJik <|||||>Hi! Indeed, if you're looking for a smaller GPT-2 model then DistilGPT-2 has half of the small GPT-2 layers. However, if you want to create your own GPT-2 model with an arbitrary number of layers, you can do so very easily with the configuration file:

```py
from transformers import GPT2Model, GPT2Config

config = GPT2Config(n_layer=2)
model = GPT2Model(config)
```

This will define a model with only two layers. Please be aware that the weights will be randomly initialized if you go down this route, whereas the weights will already be trained if you use DistilGPT-2.<|||||>Thank you guys! Really helped me a lot. I changed the GPT2Config as @LysandreJik told me and loaded pretrained weight to those layers :)
