classification, multiple choice classification and question answering.
DistilBERT
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,
Victor Sanh et al.
Same as BERT but smaller. Trained by distillation of the pretrained BERT model, meaning it's been trained to predict
the same probabilities as the larger model. The actual objective is a combination of:
finding the same probabilities as the teacher model
predicting the masked tokens correctly (but no next-sentence objective)
a cosine similarity between the hidden states of the student and the teacher model
The library provides a version of the model for masked language modeling, token classification, sentence classification
and question answering.
ConvBERT
ConvBERT: Improving BERT with Span-based Dynamic Convolution, Zihang Jiang,
