the best way to cut the upper layers
Salut, 

what would be the best way to cut the upper layers of transformers (from 12 cutting 5 upper layer results 7 layers model for the use)?

Best regards
Hi, don't know which model you are using so I can't answer precisely but here is the general workflow:
1. load the relevant pretrained configuration with `config = config_class.from_pretrained('your-model-of-interest')`
2. Reduce the number of layers in the configuration with for example: `config.num_hidden_layers = 5` (here you have to check the correct attribute for your model).
3. Use the modified config to build and instantiate your model: `model = model_class.from_pretrained('your-model-of-interest', config=config)`.

Pretty easy, isn't it?<|||||>>>Pretty easy, isn't it?

indeed!
<|||||>```
config = XLNetConfig.from_pretrained('xlnet-base-cased')
config.num_hidden_layers = 3
```
raised this error

`AttributeError: can't set attribute`<|||||>config.n_layer = 3 does it work<|||||>> Hi, don't know which model you are using so I can't answer precisely but here is the general workflow:
> 
>     1. load the relevant pretrained configuration with `config = config_class.from_pretrained('your-model-of-interest')`
> 
>     2. Reduce the number of layers in the configuration with for example: `config.num_hidden_layers = 5` (here you have to check the correct attribute for your model).
> 
>     3. Use the modified config to build and instantiate your model: `model = model_class.from_pretrained('your-model-of-interest', config=config)`.
> 
> 
> Pretty easy, isn't it?

I assume this gives the upper 5 layers. Is there a way to get the lower 5 layers ? <|||||>it's kind of annoying and non-intuitive, but @cherepanovic, the reason why you're seeing this message is that there are several parameters in many of the configs that are not parameters, but properties. I suppose the authors did this to emphasize that they should not be changed after the model is initialized? Not sure. But here they are:
```
    @property
    def max_position_embeddings(self):
        return self.n_positions

    @property
    def hidden_size(self):
        return self.n_embd

    @property
    def num_attention_heads(self):
        return self.n_head

    @property
    def num_hidden_layers(self):
        return self.n_layer
```

You can change these, after initialization, by referring to the actual parameters that the property returns.

IMO it certainly isn't "pretty easy", as doubly-named parameters/properties is kinda poor practice, and it would've been easy from a coding perspective to put getters and setters in there as well.