transformers pytorch safetensors deberta-v2 text-classification deberta-v3-base deberta-v3 deberta nli natural-language-inference multitask multi-task pipeline extreme-multi-task extreme-mtl tasksource zero-shot rlhf zero-shot-classification en dataset:glue dataset:nyu-mll/multi_nli dataset:multi_nli dataset:super_glue dataset:anli dataset:tasksource/babi_nli dataset:sick dataset:snli dataset:scitail dataset:OpenAssistant/oasst1 dataset:universal_dependencies dataset:hans dataset:qbao775/PARARULE-Plus dataset:alisawuffles/WANLI dataset:metaeval/recast dataset:sileod/probability_words_nli dataset:joey234/nan-nli dataset:pietrolesci/nli_fever dataset:pietrolesci/breaking_nli dataset:pietrolesci/conj_nli dataset:pietrolesci/fracas dataset:pietrolesci/dialogue_nli dataset:pietrolesci/mpe dataset:pietrolesci/dnc dataset:pietrolesci/gpt3_nli dataset:pietrolesci/recast_white dataset:pietrolesci/joci dataset:martn-nguyen/contrast_nli dataset:pietrolesci/robust_nli dataset:pietrolesci/robust_nli_is_sd dataset:pietrolesci/robust_nli_li_ts dataset:pietrolesci/gen_debiased_nli dataset:pietrolesci/add_one_rte dataset:metaeval/imppres dataset:pietrolesci/glue_diagnostics dataset:hlgd dataset:PolyAI/banking77 dataset:paws dataset:quora dataset:medical_questions_pairs dataset:conll2003 dataset:nlpaueb/finer-139 dataset:Anthropic/hh-rlhf dataset:Anthropic/model-written-evals dataset:truthful_qa dataset:nightingal3/fig-qa dataset:tasksource/bigbench dataset:blimp dataset:cos_e dataset:cosmos_qa dataset:dream dataset:openbookqa dataset:qasc dataset:quartz dataset:quail dataset:head_qa dataset:sciq dataset:social_i_qa dataset:wiki_hop dataset:wiqa dataset:piqa dataset:hellaswag dataset:pkavumba/balanced-copa dataset:12ml/e-CARE dataset:art dataset:tasksource/mmlu dataset:winogrande dataset:codah dataset:ai2_arc dataset:definite_pronoun_resolution dataset:swag dataset:math_qa dataset:metaeval/utilitarianism dataset:mteb/amazon_counterfactual dataset:SetFit/insincere-questions dataset:SetFit/toxic_conversations dataset:turingbench/TuringBench dataset:trec dataset:tals/vitaminc dataset:hope_edi dataset:strombergnlp/rumoureval_2019 dataset:ethos dataset:tweet_eval dataset:discovery dataset:pragmeval dataset:silicone dataset:lex_glue dataset:papluca/language-identification dataset:imdb dataset:rotten_tomatoes dataset:ag_news dataset:yelp_review_full dataset:financial_phrasebank dataset:poem_sentiment dataset:dbpedia_14 dataset:amazon_polarity dataset:app_reviews dataset:hate_speech18 dataset:sms_spam dataset:humicroedit dataset:snips_built_in_intents dataset:banking77 dataset:hate_speech_offensive dataset:yahoo_answers_topics dataset:pacovaldez/stackoverflow-questions dataset:zapsdcn/hyperpartisan_news dataset:zapsdcn/sciie dataset:zapsdcn/citation_intent dataset:go_emotions dataset:allenai/scicite dataset:liar dataset:relbert/lexical_relation_classification dataset:metaeval/linguisticprobing dataset:tasksource/crowdflower dataset:metaeval/ethics dataset:emo dataset:google_wellformed_query dataset:tweets_hate_speech_detection dataset:has_part dataset:wnut_17 dataset:ncbi_disease dataset:acronym_identification dataset:jnlpba dataset:species_800 dataset:SpeedOfMagic/ontonotes_english dataset:blog_authorship_corpus dataset:launch/open_question_type dataset:health_fact dataset:commonsense_qa dataset:mc_taco dataset:ade_corpus_v2 dataset:prajjwal1/discosense dataset:circa dataset:PiC/phrase_similarity dataset:copenlu/scientific-exaggeration-detection dataset:quarel dataset:mwong/fever-evidence-related dataset:numer_sense dataset:dynabench/dynasent dataset:raquiba/Sarcasm_News_Headline dataset:sem_eval_2010_task_8 dataset:demo-org/auditor_review dataset:medmcqa dataset:aqua_rat dataset:RuyuanWan/Dynasent_Disagreement dataset:RuyuanWan/Politeness_Disagreement dataset:RuyuanWan/SBIC_Disagreement dataset:RuyuanWan/SChem_Disagreement dataset:RuyuanWan/Dilemmas_Disagreement dataset:lucasmccabe/logiqa dataset:wiki_qa dataset:metaeval/cycic_classification dataset:metaeval/cycic_multiplechoice dataset:metaeval/sts-companion dataset:metaeval/commonsense_qa_2.0 dataset:metaeval/lingnli dataset:metaeval/monotonicity-entailment dataset:metaeval/arct dataset:metaeval/scinli dataset:metaeval/naturallogic dataset:onestop_qa dataset:demelin/moral_stories dataset:corypaik/prost dataset:aps/dynahate dataset:metaeval/syntactic-augmentation-nli dataset:metaeval/autotnli dataset:lasha-nlp/CONDAQA dataset:openai/webgpt_comparisons dataset:Dahoas/synthetic-instruct-gptj-pairwise dataset:metaeval/scruples dataset:metaeval/wouldyourather dataset:sileod/attempto-nli dataset:metaeval/defeasible-nli dataset:metaeval/help-nli dataset:metaeval/nli-veridicality-transitivity dataset:metaeval/natural-language-satisfiability dataset:metaeval/lonli dataset:tasksource/dadc-limit-nli dataset:ColumbiaNLP/FLUTE dataset:metaeval/strategy-qa dataset:openai/summarize_from_feedback dataset:tasksource/folio dataset:metaeval/tomi-nli dataset:metaeval/avicenna dataset:stanfordnlp/SHP dataset:GBaker/MedQA-USMLE-4-options-hf dataset:GBaker/MedQA-USMLE-4-options dataset:sileod/wikimedqa dataset:declare-lab/cicero dataset:amydeng2000/CREAK dataset:metaeval/mutual dataset:inverse-scaling/NeQA dataset:inverse-scaling/quote-repetition dataset:inverse-scaling/redefine-math dataset:tasksource/puzzte dataset:metaeval/implicatures dataset:race dataset:metaeval/spartqa-yn dataset:metaeval/spartqa-mchoice dataset:metaeval/temporal-nli dataset:metaeval/ScienceQA_text_only dataset:AndyChiang/cloth dataset:metaeval/logiqa-2.0-nli dataset:tasksource/oasst1_dense_flat dataset:metaeval/boolq-natural-perturbations dataset:metaeval/path-naturalness-prediction dataset:riddle_sense dataset:Jiangjie/ekar_english dataset:metaeval/implicit-hate-stg1 dataset:metaeval/chaos-mnli-ambiguity dataset:IlyaGusev/headline_cause dataset:metaeval/race-c dataset:metaeval/equate dataset:metaeval/ambient dataset:AndyChiang/dgen dataset:metaeval/clcd-english dataset:civil_comments dataset:metaeval/acceptability-prediction dataset:maximedb/twentyquestions dataset:metaeval/counterfactually-augmented-snli dataset:tasksource/I2D2 dataset:sileod/mindgames dataset:metaeval/counterfactually-augmented-imdb dataset:metaeval/cnli dataset:metaeval/reclor dataset:tasksource/oasst1_pairwise_rlhf_reward dataset:tasksource/zero-shot-label-nli dataset:webis/args_me dataset:webis/Touche23-ValueEval dataset:tasksource/starcon dataset:tasksource/ruletaker dataset:lighteval/lsat_qa dataset:tasksource/ConTRoL-nli dataset:tasksource/tracie dataset:tasksource/sherliic dataset:tasksource/sen-making dataset:tasksource/winowhy dataset:mediabiasgroup/mbib-base dataset:tasksource/robustLR dataset:CLUTRR/v1 dataset:tasksource/logical-fallacy dataset:tasksource/parade dataset:tasksource/cladder dataset:tasksource/subjectivity dataset:tasksource/MOH dataset:tasksource/VUAC dataset:tasksource/TroFi dataset:sharc_modified dataset:tasksource/conceptrules_v2 dataset:tasksource/disrpt dataset:conll2000 dataset:DFKI-SLT/few-nerd dataset:tasksource/com2sense dataset:tasksource/scone dataset:tasksource/winodict dataset:tasksource/fool-me-twice dataset:tasksource/monli dataset:tasksource/corr2cause dataset:tasksource/apt dataset:zeroshot/twitter-financial-news-sentiment dataset:tasksource/icl-symbol-tuning-instruct dataset:tasksource/SpaceNLI dataset:sihaochen/propsegment dataset:HannahRoseKirk/HatemojiBuild dataset:tasksource/regset dataset:lmsys/chatbot_arena_conversations dataset:tasksource/nlgraph arxiv:2301.05948 license:apache-2.0 model-index autotrain_compatible endpoints_compatible region:us
---
datasets:
- glue
- nyu-mll/multi_nli
- multi_nli
- super_glue
- anli
- tasksource/babi_nli
- sick
- snli
- scitail
- OpenAssistant/oasst1
- universal_dependencies
- hans
- qbao775/PARARULE-Plus
- alisawuffles/WANLI
- metaeval/recast
- sileod/probability_words_nli
- joey234/nan-nli
- pietrolesci/nli_fever
- pietrolesci/breaking_nli
- pietrolesci/conj_nli
- pietrolesci/fracas
- pietrolesci/dialogue_nli
- pietrolesci/mpe
- pietrolesci/dnc
- pietrolesci/gpt3_nli
- pietrolesci/recast_white
- pietrolesci/joci
- martn-nguyen/contrast_nli
- pietrolesci/robust_nli
- pietrolesci/robust_nli_is_sd
- pietrolesci/robust_nli_li_ts
- pietrolesci/gen_debiased_nli
- pietrolesci/add_one_rte
- metaeval/imppres
- pietrolesci/glue_diagnostics
- hlgd
- PolyAI/banking77
- paws
- quora
- medical_questions_pairs
- conll2003
- nlpaueb/finer-139
- Anthropic/hh-rlhf
- Anthropic/model-written-evals
- truthful_qa
- nightingal3/fig-qa
- tasksource/bigbench
- blimp
- cos_e
- cosmos_qa
- dream
- openbookqa
- qasc
- quartz
- quail
- head_qa
- sciq
- social_i_qa
- wiki_hop
- wiqa
- piqa
- hellaswag
- pkavumba/balanced-copa
- 12ml/e-CARE
- art
- tasksource/mmlu
- winogrande
- codah
- ai2_arc
- definite_pronoun_resolution
- swag
- math_qa
- metaeval/utilitarianism
- mteb/amazon_counterfactual
- SetFit/insincere-questions
- SetFit/toxic_conversations
- turingbench/TuringBench
- trec
- tals/vitaminc
- hope_edi
- strombergnlp/rumoureval_2019
- ethos
- tweet_eval
- discovery
- pragmeval
- silicone
- lex_glue
- papluca/language-identification
- imdb
- rotten_tomatoes
- ag_news
- yelp_review_full
- financial_phrasebank
- poem_sentiment
- dbpedia_14
- amazon_polarity
- app_reviews
- hate_speech18
- sms_spam
- humicroedit
- snips_built_in_intents
- banking77
- hate_speech_offensive
- yahoo_answers_topics
- pacovaldez/stackoverflow-questions
- zapsdcn/hyperpartisan_news
- zapsdcn/sciie
- zapsdcn/citation_intent
- go_emotions
- allenai/scicite
- liar
- relbert/lexical_relation_classification
- metaeval/linguisticprobing
- tasksource/crowdflower
- metaeval/ethics
- emo
- google_wellformed_query
- tweets_hate_speech_detection
- has_part
- wnut_17
- ncbi_disease
- acronym_identification
- jnlpba
- species_800
- SpeedOfMagic/ontonotes_english
- blog_authorship_corpus
- launch/open_question_type
- health_fact
- commonsense_qa
- mc_taco
- ade_corpus_v2
- prajjwal1/discosense
- circa
- PiC/phrase_similarity
- copenlu/scientific-exaggeration-detection
- quarel
- mwong/fever-evidence-related
- numer_sense
- dynabench/dynasent
- raquiba/Sarcasm_News_Headline
- sem_eval_2010_task_8
- demo-org/auditor_review
- medmcqa
- aqua_rat
- RuyuanWan/Dynasent_Disagreement
- RuyuanWan/Politeness_Disagreement
- RuyuanWan/SBIC_Disagreement
- RuyuanWan/SChem_Disagreement
- RuyuanWan/Dilemmas_Disagreement
- lucasmccabe/logiqa
- wiki_qa
- metaeval/cycic_classification
- metaeval/cycic_multiplechoice
- metaeval/sts-companion
- metaeval/commonsense_qa_2.0
- metaeval/lingnli
- metaeval/monotonicity-entailment
- metaeval/arct
- metaeval/scinli
- metaeval/naturallogic
- onestop_qa
- demelin/moral_stories
- corypaik/prost
- aps/dynahate
- metaeval/syntactic-augmentation-nli
- metaeval/autotnli
- lasha-nlp/CONDAQA
- openai/webgpt_comparisons
- Dahoas/synthetic-instruct-gptj-pairwise
- metaeval/scruples
- metaeval/wouldyourather
- sileod/attempto-nli
- metaeval/defeasible-nli
- metaeval/help-nli
- metaeval/nli-veridicality-transitivity
- metaeval/natural-language-satisfiability
- metaeval/lonli
- tasksource/dadc-limit-nli
- ColumbiaNLP/FLUTE
- metaeval/strategy-qa
- openai/summarize_from_feedback
- tasksource/folio
- metaeval/tomi-nli
- metaeval/avicenna
- stanfordnlp/SHP
- GBaker/MedQA-USMLE-4-options-hf
- GBaker/MedQA-USMLE-4-options
- sileod/wikimedqa
- declare-lab/cicero
- amydeng2000/CREAK
- metaeval/mutual
- inverse-scaling/NeQA
- inverse-scaling/quote-repetition
- inverse-scaling/redefine-math
- tasksource/puzzte
- metaeval/implicatures
- race
- metaeval/spartqa-yn
- metaeval/spartqa-mchoice
- metaeval/temporal-nli
- metaeval/ScienceQA_text_only
- AndyChiang/cloth
- metaeval/logiqa-2.0-nli
- tasksource/oasst1_dense_flat
- metaeval/boolq-natural-perturbations
- metaeval/path-naturalness-prediction
- riddle_sense
- Jiangjie/ekar_english
- metaeval/implicit-hate-stg1
- metaeval/chaos-mnli-ambiguity
- IlyaGusev/headline_cause
- metaeval/race-c
- metaeval/equate
- metaeval/ambient
- AndyChiang/dgen
- metaeval/clcd-english
- civil_comments
- metaeval/acceptability-prediction
- maximedb/twentyquestions
- metaeval/counterfactually-augmented-snli
- tasksource/I2D2
- sileod/mindgames
- metaeval/counterfactually-augmented-imdb
- metaeval/cnli
- metaeval/reclor
- tasksource/oasst1_pairwise_rlhf_reward
- tasksource/zero-shot-label-nli
- webis/args_me
- webis/Touche23-ValueEval
- tasksource/starcon
- tasksource/ruletaker
- lighteval/lsat_qa
- tasksource/ConTRoL-nli
- tasksource/tracie
- tasksource/sherliic
- tasksource/sen-making
- tasksource/winowhy
- mediabiasgroup/mbib-base
- tasksource/robustLR
- CLUTRR/v1
- tasksource/logical-fallacy
- tasksource/parade
- tasksource/cladder
- tasksource/subjectivity
- tasksource/MOH
- tasksource/VUAC
- tasksource/TroFi
- sharc_modified
- tasksource/conceptrules_v2
- tasksource/disrpt
- conll2000
- DFKI-SLT/few-nerd
- tasksource/com2sense
- tasksource/scone
- tasksource/winodict
- tasksource/fool-me-twice
- tasksource/monli
- tasksource/corr2cause
- tasksource/apt
- zeroshot/twitter-financial-news-sentiment
- tasksource/icl-symbol-tuning-instruct
- tasksource/SpaceNLI
- sihaochen/propsegment
- HannahRoseKirk/HatemojiBuild
- tasksource/regset
- tasksource/babi_nli
- lmsys/chatbot_arena_conversations
- tasksource/nlgraph
language: en
library_name: transformers
license: apache-2.0
metrics:
- accuracy
pipeline_tag: zero-shot-classification
tags:
- deberta-v3-base
- deberta-v3
- deberta
- text-classification
- nli
- natural-language-inference
- multitask
- multi-task
- pipeline
- extreme-multi-task
- extreme-mtl
- tasksource
- zero-shot
- rlhf
model-index:
- name: deberta-v3-base-tasksource-nli
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: glue
      type: glue
      config: rte
      split: validation
    metrics:
    - type: accuracy
      value: 0.89
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: anli-r3
      type: anli
      config: plain_text
      split: validation
    metrics:
    - type: accuracy
      value: 0.52
      name: Accuracy
---

# Model Card for DeBERTa-v3-base-tasksource-nli

---
**NOTE**

Deprecated: use https://huggingface.co/tasksource/deberta-small-long-nli for longer context and better accuracy.

---

This is [DeBERTa-v3-base](https://hf.co/microsoft/deberta-v3-base) fine-tuned with multi-task learning on 600+ tasks of the [tasksource collection](https://github.com/sileod/tasksource/).
This checkpoint has strong zero-shot validation performance on many tasks (e.g. 70% on WNLI), and can be used for:
- Zero-shot entailment-based classification for arbitrary labels [ZS].
- Natural language inference [NLI]
- Hundreds of previous tasks with tasksource-adapters [TA].
- Further fine-tuning on a new task or tasksource task (classification, token classification or multiple-choice) [FT].

# [ZS] Zero-shot classification pipeline
```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="sileod/deberta-v3-base-tasksource-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)
```
NLI training data of this model includes [label-nli](https://huggingface.co/datasets/tasksource/zero-shot-label-nli), a NLI dataset specially constructed to improve this kind of zero-shot classification.

# [NLI] Natural language inference pipeline

```python
from transformers import pipeline
pipe = pipeline("text-classification",model="sileod/deberta-v3-base-tasksource-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)
# [{'label': 'neutral', 'score': 0.9952911138534546}]
```

# [TA] Tasksource-adapters: 1 line access to hundreds of tasks 

```python
# !pip install tasknet
import tasknet as tn
pipe = tn.load_pipeline('sileod/deberta-v3-base-tasksource-nli','glue/sst2') # works for 500+ tasksource tasks
pipe(['That movie was great !', 'Awful movie.'])
# [{'label': 'positive', 'score': 0.9956}, {'label': 'negative', 'score': 0.9967}]
```
The list of tasks is available in model config.json.
This is more efficient than ZS since it requires only one forward pass per example, but it is less flexible.


# [FT] Tasknet: 3 lines fine-tuning

```python
# !pip install tasknet
import tasknet as tn
hparams=dict(model_name='sileod/deberta-v3-base-tasksource-nli', learning_rate=2e-5)
model, trainer = tn.Model_Trainer([tn.AutoTask("glue/rte")], hparams)
trainer.train()
```

## Evaluation
This model ranked 1st among all models with the microsoft/deberta-v3-base architecture according to the IBM model recycling evaluation.
https://ibm.github.io/model-recycling/

### Software and training details

The model was trained on 600 tasks for 200k steps with a batch size of 384 and a peak learning rate of 2e-5. Training took 15 days on Nvidia A30 24GB gpu.
This is the shared model with the MNLI classifier on top. Each task had a specific CLS embedding, which is dropped 10% of the time to facilitate model use without it. All multiple-choice model used the same classification layers. For classification tasks, models shared weights if their labels matched.


https://github.com/sileod/tasksource/ \
https://github.com/sileod/tasknet/ \
Training code: https://colab.research.google.com/drive/1iB4Oxl9_B5W3ZDzXoWJN-olUbqLBxgQS?usp=sharing

# Citation

More details on this [article:](https://arxiv.org/abs/2301.05948) 
```
@article{sileo2023tasksource,
  title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation},
  author={Sileo, Damien},
  url= {https://arxiv.org/abs/2301.05948},
  journal={arXiv preprint arXiv:2301.05948},
  year={2023}
}
```


# Model Card Contact

damien.sileo@inria.fr


</details>