added models

README.md

@@ -0,0 +1,106 @@
+---
+datasets:
+- squad_v2
+license: cc-by-4.0
+
+---
+
+# electra-base for QA
+
+## Overview
+**Language model:** electra-base  
+**Language:** English  
+**Downstream-task:** Extractive QA  
+**Training data:** SQuAD 2.0  
+**Eval data:** SQuAD 2.0  
+**Code:**  See [example](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering.py) in [FARM](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering.py)  
+**Infrastructure**: 1x Tesla v100
+
+## Hyperparameters
+
+```
+seed=42
+batch_size = 32
+n_epochs = 5
+base_LM_model = "google/electra-base-discriminator"
+max_seq_len = 384
+learning_rate = 1e-4
+lr_schedule = LinearWarmup
+warmup_proportion = 0.1
+doc_stride=128
+max_query_length=64
+```
+
+## Performance
+Evaluated on the SQuAD 2.0 dev set with the [official eval script](https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/).
+```
+"exact": 77.30144024256717,
+ "f1": 81.35438272008543,
+ "total": 11873,
+ "HasAns_exact": 74.34210526315789,
+ "HasAns_f1": 82.45961302894314,
+ "HasAns_total": 5928,
+ "NoAns_exact": 80.25231286795626,
+ "NoAns_f1": 80.25231286795626,
+ "NoAns_total": 5945
+```
+
+## Usage
+
+### In Transformers
+```python
+from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
+
+model_name = "deepset/electra-base-squad2"
+
+# a) Get predictions
+nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
+QA_input = {
+    'question': 'Why is model conversion important?',
+    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
+}
+res = nlp(QA_input)
+
+# b) Load model & tokenizer
+model = AutoModelForQuestionAnswering.from_pretrained(model_name)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+```
+
+### In FARM
+
+```python
+from farm.modeling.adaptive_model import AdaptiveModel
+from farm.modeling.tokenization import Tokenizer
+from farm.infer import Inferencer
+
+model_name = "deepset/electra-base-squad2"
+
+# a) Get predictions
+nlp = Inferencer.load(model_name, task_type="question_answering")
+QA_input = [{"questions": ["Why is model conversion important?"],
+             "text": "The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks."}]
+res = nlp.inference_from_dicts(dicts=QA_input)
+
+# b) Load model & tokenizer
+model = AdaptiveModel.convert_from_transformers(model_name, device="cpu", task_type="question_answering")
+tokenizer = Tokenizer.load(model_name)
+```
+
+### In haystack
+For doing QA at scale (i.e. many docs instead of single paragraph), you can load the model also in [haystack](https://github.com/deepset-ai/haystack/):
+```python
+reader = FARMReader(model_name_or_path="deepset/electra-base-squad2")
+# or
+reader = TransformersReader(model="deepset/electra-base-squad2",tokenizer="deepset/electra-base-squad2")
+```
+
+
+## Authors
+Vaishali Pal `vaishali.pal [at] deepset.ai`
+Branden Chan: `branden.chan [at] deepset.ai`
+Timo Möller: `timo.moeller [at] deepset.ai`
+Malte Pietsch: `malte.pietsch [at] deepset.ai`
+Tanay Soni: `tanay.soni [at] deepset.ai`
+
+Note:
+Borrowed this model from Haystack model repo for adding tensorflow model.

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "ElectraForQuestionAnswering"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "embedding_size": 768,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "language": "english",
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "electra",
+  "name": "Electra",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "summary_activation": "gelu",
+  "summary_last_dropout": 0,
+  "summary_type": "first",
+  "summary_use_proj": true,
+  "transformers_version": "4.19.0.dev0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

convert_pytorch_to_flax.py

@@ -0,0 +1,3 @@
+from transformers import FlaxAutoModelForQuestionAnswering
+model = FlaxAutoModelForQuestionAnswering.from_pretrained("./", from_pt=True)
+model.save_pretrained("./")

convert_pytorch_to_tensorflow.py

@@ -0,0 +1,3 @@
+from transformers import TFAutoModelForQuestionAnswering
+model = TFAutoModelForQuestionAnswering.from_pretrained("./", from_pt=True)
+model.save_pretrained("./")

flax_model.msgpack

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ff9a83a3413c0d57da7be38f7a498a5bd998f4ff8c397a9651110ba1250b94e
+size 435579889

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33abdbab680c8bae8707320df17c5d4fba5969adb465e81029905fa0284d3d3e
+size 435618605

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6323b685ea8c02486fc1a7f0df7875bd8ee320407853674bc1c1abb739d00d88
+size 435858448

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "model_max_length": 512, "special_tokens_map_file": "/home/vaishali/Documents/deepset/electra-english-qa-tutorial_epochs5/special_tokens_map.json", "full_tokenizer_file": null}