readme: # Transferring Monolingual Model to Low-Resource Language: The Case Of Tigrinya:
  
  
  
  ## Proposed Method:
  <img src="data/proposed.png" height = "330" width ="760" >
  
  The proposed method transfers a mono-lingual Transformer model into new target language at lexical level by learning new token embeddings. All implementation in this repo uses XLNet as a source Transformer model, however, other Transformer models can also be used similarly. 
  
  
  ## Main files:
  All files are IPython Notebook files which can be excuted simply in Google Colab.
  
   - train.ipynb : Fine-tunes XLNet (mono-lingual transformer) on new target language (Tigrinya) sentiment analysis dataset.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bSSrKE-TSphUyrNB2UWhFI-Bkoz0a5l0?usp=sharing)
   
   - test.ipynb : Evaluates the fine-tuned model on test data.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/17R1lvRjxILVNk971vzZT79o2OodwaNIX?usp=sharing)
   
   - token_embeddings.ipynb : Trains a word2vec token embeddings for Tigrinya language.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1hCtetAllAjBw28EVQkJFpiKdFtXmuxV7?usp=sharing)
   
   - process_Tigrinya_comments.ipynb : Extracts Tigrinya comments from mixed language contents.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1-ndLlBV-iLZNBW3Z8OfKAqUUCjvGbdZU?usp=sharing)
   
   - extract_YouTube_comments.ipynb : Downloads available comments from a YouTube channel ID.  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1b7G85wHKe18y45JIDtvDJdO5dOkRmDdp?usp=sharing)
   
   - auto_labelling.ipynb : Automatically labels Tigrinya comments in to positive or negative sentiments based on [Emoji's sentiment](http://kt.ijs.si/data/Emoji_sentiment_ranking/).    [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wnZf7CBBCIr966vRUITlxKCrANsMPpV7?usp=sharing)
   
   
  ## Tigrinya Tokenizer: 
  
  A [sentencepiece](https://github.com/google/sentencepiece) based tokenizer for Tigrinya has been released to the public and can be accessed as in the following:
  
       
       from transformers import AutoTokenizer
       tokenizer = AutoTokenizer.from_pretrained("abryee/TigXLNet")
       tokenizer.tokenize("ዋዋዋው እዛ ፍሊም ካብተን ዘድንቀን ሓንቲ ኢያ ሞ ብጣዕሚ ኢና ነመስግን ሓንቲ ክብላ ደልየ ዘሎኹ ሓደራኣኹም ኣብ ጊዜኹም ተረክቡ")
      
  
   ## TigXLNet:
   A new general purpose transformer model for low-resource language Tigrinya is also released to the public and be accessed as in the following:
      
      from transformers import AutoConfig, AutoModel
      config = AutoConfig.from_pretrained("abryee/TigXLNet")
      config.d_head = 64
      model = AutoModel.from_pretrained("abryee/TigXLNet", config=config)
   
   ## Evaluation:
   
   The proposed method is evaluated using two datasets:
    - A newly created sentiment analysis dataset for low-resource language (Tigriyna). 
     
    <table>
     <tr>
      <td> <table>
      <thead>
          <tr>
              <th><sub>Models</sub></th>
              <th><sub>Configuration</sub></th>
              <th><sub>F1-Score</sub></th>
          </tr>
      </thead>
      <tbody>
          <tr>
              <td rowspan=3><sub>BERT</sub></td>
              <td rowspan=1><sub>+Frozen BERT weights</sub></td>
              <td><sub>54.91</sub></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Random embeddings</sub></td>
              <td><sub>74.26</sub></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Frozen token embeddings</sub></td>
              <td><sub>76.35</sub></td>
          </tr>     
          <tr>
              <td rowspan=3><sub>mBERT</sub></td>
              <td rowspan=1><sub>+Frozen mBERT weights</sub></td>
              <td><sub>57.32</sub></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Random embeddings</sub></td>
              <td><sub>76.01</sub></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Frozen token embeddings</sub></td>
              <td><sub>77.51</sub></td>
          </tr>        
          <tr>
              <td rowspan=3><sub>XLNet</sub></td>
              <td rowspan=1><sub>+Frozen XLNet weights</sub></td>
              <td><strong><sub>68.14</sub></strong></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Random embeddings</sub></td>
              <td><strong><sub>77.83</sub></strong></td>
          </tr>
          <tr>
              <td rowspan=1><sub>+Frozen token embeddings</sub></td>
              <td><strong><sub>81.62</sub></strong></td>
          </tr>
      </tbody>
  </table> </td>
        <td><img src="data/effect_of_dataset_size.png" alt="3" width = 480px height = 280px></td>
    </tr>
   </table>
  
    
          
    - Cross-lingual Sentiment dataset ([CLS](https://zenodo.org/record/3251672#.Xs65VzozbIU)).
    
    
    <table>
      <thead>