readme: ---
  datasets:
  - mnli
  - xnli
  tags:
  - sentence-similarity
  - transformers
  - text-classification
  - zero-shot-classification
  - salesken
  - hindi
  - cross-lingual
  inference: false
  ---
  
  # XLM-R Base
  
  A multilingual model is pre-trained on text coming from a mix of languages. We will look at a multilingual model called XLM-R from Facebook also called as Cross Lingual Model - Roberta. Unlike BERT which was pre-trained on English Wikipedia and BookCorpus, XLM-R was pre-trained on Wikipedia and Common Crawl data from 100 different languages. A single model trained on 100 different languages. 
  * So, there is a single shared vocabulary with 250K tokens to cover all 100 languages.
  * There is no special marker which suggests which language it is.
  * It wasn't trained with parallel data (i.e. same sentence in multiple languages)
  
  # Cross Lingual Transfer (Zero-shot transfer)
  
  Say you are building a classifier which will classify Hindi comments toxic or non-toxic. But say you have a training dataset in English of 225 K labeled comments called "Wikipedia Toxic Comments". The Cross Lingual Transfer of XLM-R will let you fine-tune XLM-R on Wikipedia Toxic Comments dataset in English, and then apply it to Hindi comments. So, XLM-R is able to take it's task specific knowledge that it learned in English and apply it on Hindi, even though we never showed it any Hindi exmaples. It's concept of transfer learning applied from one language to another which is called Cross Lingual Transfer. The same could be done on any other low-resource languages
  
  We will see that training XLM-R purely on ~400K English samples acutally yields better results than fine-tuning a monolingual Hindi model on a much smaller Hindi dataset. This is also referred to as Zero-Shot Learning or Cross-Lingual Transfer
  
  # Our Approach -
  
  We will use MNLI that provides us with a large number of English training examples to fine-tune XLM-R on the general task of NLI. XNLI will provide us with a small number of NLI test examples in different languages for us it will be Hindi. MNLI (Multi-genre NLI) has 392,000 training examples and 20,000 development examples and 20,000 test examples. On the other hand XNLI is a small subset of examples from MNLI dataset which have been human-translated to 14 different languages and yes Hindi is a part of it. For each language we have 5,000 test set sentence pairs and 2,500 development set sentence pairs
  
  # Fine-tuning Hyperparameters -
  
  learning rate : 5e-6 with linear learning rate scheduling over total steps
  epochs = 2
  batch_size = 32
  GPU : Tesla T4
  
  # Using this model
  This model can be used using Huggingface Transformers. I have created a pipeline for a sentence pair classification. Hope it will be useful.
  
  ```python
  !pip3 install transformers
  
  import transformers
  import torch
  
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
    
  tokenizer = AutoTokenizer.from_pretrained("salesken/xlm-roberta-base-finetuned-mnli-cross-lingual-transfer")
  
  model = AutoModelForSequenceClassification.from_pretrained("salesken/xlm-roberta-base-finetuned-mnli-cross-lingual-transfer")
  
  device = torch.device('cuda')
  print('GPU: ', torch.cuda.get_device_name(0))
  
  model.to(device)
  
  # Here I am giving the sentence pairs and also the ground truth values (1 : Similar, 0 : NotSimilar)
  # we have purposely used Hindi-Hindi , Hindi-English, Transliterated Hindi-Engish, Kannada and Spanish and also code-switching of languages
  # You can try your own pairs and languages
  
  u_premise = ['क्या आपका बेटा गणित में रूचि रखता है',
              'क्या आपका बेटा गणित में रूचि रखता है',
              'क्या आपका बेटा गणित में रूचि रखता है',
              'मैं त्योहार के दौरान घर जा रहा हूँ',
               'क्या आपका बेटा गणित में रूचि रखता है',
              'hum ek cross lingual model banane ka soch rahe hai',
              'we are planning to build a cross-lingual language model',
              'Hi. A very good morning to all of you. Lets get started. ',
              'शुभ प्रभात',
              'how are you doing today sir ?',
              'how are you doing today sir ?',
              'Argentina and Brazil will play Copa America finals this weekend',
               'I want to visit Spain and Portugal next year'
              ]
  
  u_hypothesis = ['क्या आपकी बेटी को विज्ञान में दिलचस्पी है',
                 'is your daughther interested in science',
                 'is your son interested in stem subjects',
                 'इस साल मैं त्योहार के दौरान घर नहीं जाऊंगा',
                  'इस साल मैं त्योहार के दौरान घर नहीं जाऊंगा',
                 'हम एक क्रॉस-लिंगुअल लैंग्वेज मॉडल बनाने की योजना बना रहे हैं',
                 'hum ek क्रॉस-लिंगुअल model bana rahe hai',
                 'शुभ प्रभात',
                 'subh prabhat to all of you',
                 'ಇಂದು ನೀವು ಹೇಗಿದ್ದೀರಿ ಸರ್?',
                 'ನನ್ನ ಸ್ನಾತಕೋತ್ತರರಿಗಾಗಿ ನಾನು ಯಂತ್ರ ಕಲಿಕೆ ಮತ್ತು ಡೇಟಾ ವಿಜ್ಞಾನವನ್ನು ಮಾಡುತ್ತಿದ್ದೇನೆ',
                 'Argentina y Brasil jugarán la final de la Copa América este fin de semana',
                 'Pue