readme: ---
  language: multilingual
  tags:
  - pytorch
  license: apache-2.0
  datasets:
  - multi_nli
  - xnli
  metrics:
  - xnli
  
  ---
  
  # mt5-large-finetuned-mnli-xtreme-xnli
  
  ## Model Description
  
  
  This model takes a pretrained large [multilingual-t5](https://github.com/google-research/multilingual-t5) (also available from [models](https://huggingface.co/google/mt5-large)) and fine-tunes it on English MNLI and the [xtreme_xnli](https://www.tensorflow.org/datasets/catalog/xtreme_xnli) training set. It is intended to be used for zero-shot text classification, inspired by [xlm-roberta-large-xnli](https://huggingface.co/joeddav/xlm-roberta-large-xnli).
  
  ## Intended Use
  
  This model is intended to be used for zero-shot text classification, especially in languages other than English. It is fine-tuned on English MNLI and the [xtreme_xnli](https://www.tensorflow.org/datasets/catalog/xtreme_xnli) training set, a multilingual NLI dataset. The model can therefore be used with any of the languages in the XNLI corpus:
  
  - Arabic
  - Bulgarian
  - Chinese
  - English
  - French
  - German
  - Greek
  - Hindi
  - Russian
  - Spanish
  - Swahili
  - Thai
  - Turkish
  - Urdu
  - Vietnamese
  
  
  As per recommendations in [xlm-roberta-large-xnli](https://huggingface.co/joeddav/xlm-roberta-large-xnli), for English-only classification, you might want to check out:
  - [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli)
  - [a distilled bart MNLI model](https://huggingface.co/models?filter=pipeline_tag%3Azero-shot-classification&search=valhalla).
  
  
  ### Zero-shot example:
  
  The model retains its text-to-text characteristic after fine-tuning. This means that our expected outputs will be text. During fine-tuning, the model learns to respond to the NLI task with a series of single token responses that map to entailment, neutral, or contradiction. The NLI task is indicated with a fixed prefix, "xnli:". 
  
  Below is an example, using PyTorch, of the model's use in a similar fashion to the `zero-shot-classification` pipeline. We use the logits from the LM output at the first token to represent confidence.
  
  ```python
  from torch.nn.functional import softmax
  from transformers import MT5ForConditionalGeneration, MT5Tokenizer
  
  model_name = "alan-turing-institute/mt5-large-finetuned-mnli-xtreme-xnli"
  
  tokenizer = MT5Tokenizer.from_pretrained(model_name)
  model = MT5ForConditionalGeneration.from_pretrained(model_name)
  model.eval()
  
  sequence_to_classify = "¿A quién vas a votar en 2020?"
  candidate_labels = ["Europa", "salud pública", "política"]
  hypothesis_template = "Este ejemplo es {}."
  
  ENTAILS_LABEL = "▁0"
  NEUTRAL_LABEL = "▁1"
  CONTRADICTS_LABEL = "▁2"
  
  label_inds = tokenizer.convert_tokens_to_ids(
      [ENTAILS_LABEL, NEUTRAL_LABEL, CONTRADICTS_LABEL])
  
  
  def process_nli(premise: str, hypothesis: str):
      """ process to required xnli format with task prefix """
      return "".join(['xnli: premise: ', premise, ' hypothesis: ', hypothesis])
  
  
  # construct sequence of premise, hypothesis pairs
  pairs = [(sequence_to_classify, hypothesis_template.format(label)) for label in
          candidate_labels]
  # format for mt5 xnli task
  seqs = [process_nli(premise=premise, hypothesis=hypothesis) for
          premise, hypothesis in pairs]
  print(seqs)
  # ['xnli: premise: ¿A quién vas a votar en 2020? hypothesis: Este ejemplo es Europa.',
  # 'xnli: premise: ¿A quién vas a votar en 2020? hypothesis: Este ejemplo es salud pública.',
  # 'xnli: premise: ¿A quién vas a votar en 2020? hypothesis: Este ejemplo es política.']
  
  inputs = tokenizer.batch_encode_plus(seqs, return_tensors="pt", padding=True)
  
  out = model.generate(**inputs, output_scores=True, return_dict_in_generate=True,
                       num_beams=1)
  
  # sanity check that our sequences are expected length (1 + start token + end token = 3)
  for i, seq in enumerate(out.sequences):
      assert len(
          seq) == 3, f"generated sequence {i} not of expected length, 3." \\\\
                     f" Actual length: {len(seq)}"
  
  # get the scores for our only token of interest
  # we'll now treat these like the output logits of a `*ForSequenceClassification` model
  scores = out.scores[0]
  
  # scores has a size of the model's vocab.
  # However, for this task we have a fixed set of labels
  # sanity check that these labels are always the top 3 scoring
  for i, sequence_scores in enumerate(scores):
      top_scores = sequence_scores.argsort()[-3:]
      assert set(top_scores.tolist()) == set(label_inds), \\\\
          f"top scoring tokens are not expected for this task." \\\\
          f" Expected: {label_inds}. Got: {top_scores.tolist()}."
  
  # cut down scores to our task labels
  scores = scores[:, label_inds]
  print(scores)
  # tensor([[-2.5697,  1.0618,  0.2088],
  #         [-5.4492, -2.1805, -0.1473],
  #         [ 2.2973,  3.7595, -0.1769]])
  
  
  # new indices of entailment and contradiction in scores
  entailment_ind = 0
  contradiction_ind = 2
  
  # we can show, per item, the entailment vs contradiction probas
  entail_vs_contra_scores = scores[:, [entailment_ind, contradiction_ind]]
  entail_vs_cont