readme: ---
  language: en
  datasets:
  - squad_v2
  license: cc-by-4.0
  ---
  
  # tinyroberta-squad2
  
  ## Overview
  **Language model:** tinyroberta-squad2  
  **Language:** English  
  **Training data:** The PILE  
  **Code:**  
  **Infrastructure**: 4x Tesla v100
  
  ## Hyperparameters
  
  ```
  batch_size = 96
  n_epochs = 4
  base_LM_model = "deepset/tinyroberta-squad2-step1"
  max_seq_len = 384
  learning_rate = 1e-4
  lr_schedule = LinearWarmup
  warmup_proportion = 0.2
  teacher = "deepset/roberta-base"
  ``` 
  
  ## Distillation
  This model was distilled using the TinyBERT approach described in [this paper](https://arxiv.org/pdf/1909.10351.pdf) and implemented in [haystack](https://github.com/deepset-ai/haystack).
  We have performed intermediate layer distillation with roberta-base as the teacher which resulted in [deepset/tinyroberta-6l-768d](https://huggingface.co/deepset/tinyroberta-6l-768d).
  This model has not been distilled for any specific task. If you are interested in using distillation to improve its performance on a downstream task, you can take advantage of haystack's new [distillation functionality](https://haystack.deepset.ai/guides/model-distillation). You can also check out [deepset/tinyroberta-squad2](https://huggingface.co/deepset/tinyroberta-squad2) for a model that is already distilled on an extractive QA downstream task.
  
  ## Usage
  
  ### In Transformers
  ```python
  from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
  
  model_name = "deepset/tinyroberta-squad2"
  
  model = AutoModelForQuestionAnswering.from_pretrained(model_name)
  tokenizer = AutoTokenizer.from_pretrained(model_name)
  ```
  
  ### In FARM
  
  ```python
  from farm.modeling.adaptive_model import AdaptiveModel
  from farm.modeling.tokenization import Tokenizer
  from farm.infer import Inferencer
  
  model_name = "deepset/tinyroberta-squad2"
  model = AdaptiveModel.convert_from_transformers(model_name, device="cpu", task_type="question_answering")
  tokenizer = Tokenizer.load(model_name)
  ```
  
  ### In haystack
  For doing QA at scale (i.e. many docs instead of single paragraph), you can load the model also in [haystack](https://github.com/deepset-ai/haystack/):
  ```python
  reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
  # or 
  reader = TransformersReader(model_name_or_path="deepset/roberta-base-squad2",tokenizer="deepset/roberta-base-squad2")
  ```
  
  
  ## Authors
  Branden Chan: `branden.chan [at] deepset.ai`
  Timo Möller: `timo.moeller [at] deepset.ai`
  Malte Pietsch: `malte.pietsch [at] deepset.ai`
  Tanay Soni: `tanay.soni [at] deepset.ai`
  Michel Bartels: `michel.bartels [at] deepset.ai`
  
  ## About us
  ![deepset logo](https://workablehr.s3.amazonaws.com/uploads/account/logo/476306/logo)
  We bring NLP to the industry via open source!  
  Our focus: Industry specific language models & large scale QA systems.  
    
  Some of our work: 
  - [German BERT (aka "bert-base-german-cased")](https://deepset.ai/german-bert)
  - [GermanQuAD and GermanDPR datasets and models (aka "gelectra-base-germanquad", "gbert-base-germandpr")](https://deepset.ai/germanquad)
  - [FARM](https://github.com/deepset-ai/FARM)
  - [Haystack](https://github.com/deepset-ai/haystack/)
  
  Get in touch:
  [Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Slack](https://haystack.deepset.ai/community/join) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://deepset.ai)
  
  By the way: [we're hiring!](http://www.deepset.ai/jobs) 