NLP, Artificial Intelligence, Machine Learning, Data Science, Transformers.
it’s an autoregressive Transformer that can reuse previously computed hidden-states to attend to longer context. ALBERT [2019]: a lighter version of BERT, where (1) Next Sentence Prediction is replaced by Sentence Order Prediction, and (2) parameter-reduction techniques are used for lower memory