NLP, Artificial Intelligence, Machine Learning, Data Science, Transformers.
ConvBERT [2019]: a better version of BERT, where self-attention blocks are replaced with new ones that leverage convolutions to better model global and local context. Funnel Transformer [2020]: a type of Transformer that gradually compresses the sequence of hidden states to a shorter one and hence