NLP, Artificial Intelligence, Machine Learning, Data Science, Transformers.
embeddings and to better align output to input. BERT [2018]: bidirectional Transformer pre-trained using a combination of Masked Language Modeling and Next Sentence Prediction objectives. It uses global attention. GPT [2018]: the first autoregressive model based on the Transformer architecture.