Question: dropping transformers layers
Hi everyone,

I came across this interesting paper: _[On the Effect of Dropping Layers of Pre-trained Transformer Models](https://arxiv.org/pdf/2004.03844.pdf)_ and I am actually trying to implement it. However, I am wondering what would be a good practice to perform _layer dropping_.

Would it be safer to just mask the unwanted layers (some sort of pruning)? Or maybe copy the wanted layers into a new model? I've also seen that you guys have a `nn_pruning` library involving some sort of `SparseTrainers`, would that be the way to go?
Would love to hear your suggestions on this one 😃 

Thanks for your help and thanks a lot for the amazing repo!
Cheers,
Jules
Hey @JulesBelveze, we generally try to keep the github issues for bugs/feature requests, and foster discussions on the [forum](https://discuss.huggingface.co) instead.

For your question regarding layer dropping, we actually have a few architectures that support it! You can search for the term `layerdrop`: https://github.com/huggingface/transformers/search?q=layerdrop

Examples of those are the BART, Marian, mBART, Pegasus family of models.

Thanks!<|||||>Hey @LysandreJik, really sorry for asking the question here.
Thanks for your answer, that's exactly what I was looking for!

Cheers