Answers.Body: <p>For pretrained models in NLP, look at BERT and RoBERTa. If you can find a language model trained on your data's superset on Huggingface, then, use that pretrained model.</p>
  <p>In order to multiclass classification, since your data is less, look at augmentations in NLP (most notably, backtranslation amongst others). Use focal loss (to handle class imbalance).</p>
  <p>Since, you are going to finetune use small learning, 1e-5. But, you will be adding your own layers also, so keep 1e-5 for the pretrained model and 1e-3 for the new layer you put.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-27T18:58:12.847Z
Answers.LastActivityDate: 2021-05-27T18:58:12.847Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I have a training dataset of 80 text documents with an average number of characters in each document of 25000 and 210 unique tags.</p>
  <p>How can I perform multi-class text classification with such a small dataset, without using the pre-trained model? If it cannot be done without a pre-trained model, then which pre-trained model should I use?</p>
  
Tags: <machine-learning><natural-language-processing><transfer-learning><pretrained-models><multiclass-classification>
Title: How to perform multi-class text classification with a dataset of 80 documents?
UserDisplayName: Utkarsh Malkoti
ThreadText: # How to perform multi-class text classification with a dataset of 80 documents?
  
  (Asked by: username_0 on 2021-05-27)
  
  I have a training dataset of 80 text documents with an average number of characters in each document of 25000 and 210 unique tags.
  
  How can I perform multi-class text classification with such a small dataset, without using the pre-trained model? If it cannot be done without a pre-trained model, then which pre-trained model should I use?
  
  **Tags:** machine-learning, natural-language-processing, transfer-learning, pretrained-models, multiclass-classification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-27. Score: 1
  
  For pretrained models in NLP, look at BERT and RoBERTa. If you can find a language model trained on your data's superset on Huggingface, then, use that pretrained model.
  
  In order to multiclass classification, since your data is less, look at augmentations in NLP (most notably, backtranslation amongst others). Use focal loss (to handle class imbalance).
  
  Since, you are going to finetune use small learning, 1e-5. But, you will be adding your own layers also, so keep 1e-5 for the pretrained model and 1e-3 for the new layer you put.
  
  
  ---
  
  