NLP, Artificial Intelligence, Machine Learning, Data Science, Transformers.
Longformer [2020]: a Transformer model replacing the attention matrices with sparse matrices for higher training efficiency. ProphetNet [2020]: a Transformer model trained with the Future N-gram Prediction objective and with a novel self-attention mechanism. ELECTRA [2020]: same as BERT but lighter