readme: ---
  language: en
  datasets:
  - c4
  - wikipedia
  metrics:
  - f1
  ---
  
  # T5-V1.1-large-rss
  This model is [T5-v1.1-large](https://huggingface.co/google/t5-v1_1-large) finetuned on RSS dataset. The model was finetuned as part of 
  ["How Optimal is Greedy Decoding for Extractive Question Answering?"](https://arxiv.org/abs/2108.05857), while the RSS pretraining method was introduced in [this paper](https://arxiv.org/pdf/2101.00438.pdf).
  
  ## Model description
  The original [T5-v1.1-large](https://huggingface.co/google/t5-v1_1-large) was only pre-trained on C4 excluding any supervised training. Our version is further trained on Rucurrent Span Selection scheme (RSS), using a sample from the dataset used to pretrain [Splinter](tau/splinter-large):
  * contexts with a span occurring more than once are detected
  * a single instance of the recurring span is maked
  * the model is trained (teacher forcing) to predict the masked span
  This training scheme naturally matches the extractive question answering task.
  
  During training time, the masked span is replaced with `<extra_id_0>` and the labels are formatted as `<extra_id_0>span<extra_id_0>`. Unlike [Splinter](tau/splinter-large), only one span is mask at a time.
  
  ## Intended uses & limitations
  This model naturally fits tasks where a span from a context is intended to be copied, like extractive question answering.
  This checkpoint is primarily aimed to be used in zero-shot setting - further fine-tuning it on an annotated dataset gives equal results to those of the original T5-v1.1-large.
  
  ### How to use
  You can use this model directly but it is recommended to format the input to be aligned with that of the training scheme and as a text-question context:
  ```python
  from transformers import  AutoModelForSeq2SeqLM, AutoTokenizer
  model = AutoModelForSeq2SeqLM.from_pretrained('tau/t5-v1_1-large-rss')
  tokenizer = AutoTokenizer.from_pretrained('tau/t5-v1_1-large-rss')
  
  passage = 'Barack Hussein Obama II is an American politician and attorney who served as the 44th president of the United States from 2009 to 2017. '
  question = 'When was Obama inaugurated?'
  text = f'Text: {passage}.\nQuestion: {question}\nAnswer:{tokenizer.additional_special_tokens[0]}.'
  encoded_input = tokenizer(text, return_tensors='pt')
  output_ids = model.generate(input_ids=encoded_input.input_ids, attention_mask=encoded_input.attention_mask,
                 eos_token_id=tokenizer.additional_special_tokens_ids[1], num_beams=1, max_length=512, min_length=3)
  tokenizer.decode(output_ids[0])
  ```
  The generated answer is then `"<pad><extra_id_0> 2009<extra_id_1>"`, while the one generated by the original [T5-v1.1-large](https://huggingface.co/google/t5-v1_1-large) is `"<pad><extra_id_0> On January 20, 2009<extra_id_1>"` - a correct yet non-extractive answer.
  
  ### Limitations and bias
  Although using the model with greedy decoding tends toward extracted outputs, is may sometimes produce non-extracted ones - may it be different casing or a whole different string (or substring) that may bear another semantic meaning.
  
  ### Pretraining
  The model was finetuned with 100,000 rss-examples for 3 epochs using Adafactor optimizer with constant learning rate of 5e-5.
  
  ## Evaluation results
  Evaluated over few-shot QA in a zero-shot setting (no finetuning on annotated examples):
  
  |Model \ Dataset| SQuAD |TriviaQA | NaturalQs | NewsQA | SearchQA | HotpotQA | BioASQ | TextbookQA| 
  |:-------------:|:-----:|:-------:|:---------:|:------:|:--------:|:--------:|:------:|:---------:| 
  |T5             | 50.4  | 61.7    | 42.1      | 19.2   | 24.0     | 43.3     | 55.5   | 17.8      | 
  |T5-rss         | 71.4  | 69.3    | 57.2      | 43.2   | 29.7     | 59.0     | 65.5   | 39.0      | 
  
  The gap between the two models diminishes as more training examples are introduced, for additional result see the [paper]((https://arxiv.org/abs/2108.05857).
  
  ### BibTeX entry and citation info
  ```bibtex
  @inproceedings{ram-etal-2021-shot,
      title = "Few-Shot Question Answering by Pretraining Span Selection",
      author = "Ram, Ori  and
        Kirstain, Yuval  and
        Berant, Jonathan  and
        Globerson, Amir  and
        Levy, Omer",
      booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
      month = aug,
      year = "2021",
      address = "Online",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/2021.acl-long.239",
      doi = "10.18653/v1/2021.acl-long.239",
      pages = "3066--3079",
  },
  @misc{castel2021optimal,
        title={How Optimal is Greedy Decoding for Extractive Question Answering?}, 
        author={Or Castel and Ori Ram and Avia Efrat and Omer Levy},
        year={2021},
        eprint={2108.05857},
        archivePrefix={arXiv},
        primaryClass={cs.CL}
  }
  
  ```
  