Funnel Transformers
# 🌟 New model addition
Funnel-Transformer

## Model description
Funnel-Transformer is a new self-attention model that gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, Funnel-Transformer usually has a higher capacity given the same FLOPs. In addition, with a decoder, Funnel-Transformer is able to recover the token-level deep representation for each token from the reduced hidden sequence, which enables standard pretraining.

<!-- Important information -->

## Open source status
Released.

* [x] the model implementation is available: (give details)
https://github.com/laiguokun/Funnel-Transformer
* [x] the model weights are available: (give details)
https://github.com/laiguokun/Funnel-Transformer
* [x] who are the authors: (mention them, if possible by @gh-username)
Zihang Dai*, Guokun Lai*, Yiming Yang, Quoc V. Le

Duplicate of #4844?<|||||>my bad. yes