transformers pytorch safetensors deberta-v2 text-classification deberta-v3-large nli natural-language-inference multitask multi-task pipeline extreme-multi-task extreme-mtl tasksource zero-shot rlhf zero-shot-classification en dataset:glue dataset:super_glue dataset:anli dataset:metaeval/babi_nli dataset:sick dataset:snli dataset:scitail dataset:hans dataset:alisawuffles/WANLI dataset:metaeval/recast dataset:sileod/probability_words_nli dataset:joey234/nan-nli dataset:pietrolesci/nli_fever dataset:pietrolesci/breaking_nli dataset:pietrolesci/conj_nli dataset:pietrolesci/fracas dataset:pietrolesci/dialogue_nli dataset:pietrolesci/mpe dataset:pietrolesci/dnc dataset:pietrolesci/gpt3_nli dataset:pietrolesci/recast_white dataset:pietrolesci/joci dataset:martn-nguyen/contrast_nli dataset:pietrolesci/robust_nli dataset:pietrolesci/robust_nli_is_sd dataset:pietrolesci/robust_nli_li_ts dataset:pietrolesci/gen_debiased_nli dataset:pietrolesci/add_one_rte dataset:metaeval/imppres dataset:pietrolesci/glue_diagnostics dataset:hlgd dataset:paws dataset:quora dataset:medical_questions_pairs dataset:conll2003 dataset:Anthropic/hh-rlhf dataset:Anthropic/model-written-evals dataset:truthful_qa dataset:nightingal3/fig-qa dataset:tasksource/bigbench dataset:bigbench dataset:blimp dataset:cos_e dataset:cosmos_qa dataset:dream dataset:openbookqa dataset:qasc dataset:quartz dataset:quail dataset:head_qa dataset:sciq dataset:social_i_qa dataset:wiki_hop dataset:wiqa dataset:piqa dataset:hellaswag dataset:pkavumba/balanced-copa dataset:12ml/e-CARE dataset:art dataset:tasksource/mmlu dataset:winogrande dataset:codah dataset:ai2_arc dataset:definite_pronoun_resolution dataset:swag dataset:math_qa dataset:metaeval/utilitarianism dataset:mteb/amazon_counterfactual dataset:SetFit/insincere-questions dataset:SetFit/toxic_conversations dataset:turingbench/TuringBench dataset:trec dataset:tals/vitaminc dataset:hope_edi dataset:strombergnlp/rumoureval_2019 dataset:ethos dataset:tweet_eval dataset:discovery dataset:pragmeval dataset:silicone dataset:lex_glue dataset:papluca/language-identification dataset:imdb dataset:rotten_tomatoes dataset:ag_news dataset:yelp_review_full dataset:financial_phrasebank dataset:poem_sentiment dataset:dbpedia_14 dataset:amazon_polarity dataset:app_reviews dataset:hate_speech18 dataset:sms_spam dataset:humicroedit dataset:snips_built_in_intents dataset:banking77 dataset:hate_speech_offensive dataset:yahoo_answers_topics dataset:pacovaldez/stackoverflow-questions dataset:zapsdcn/hyperpartisan_news dataset:zapsdcn/sciie dataset:zapsdcn/citation_intent dataset:go_emotions dataset:scicite dataset:liar dataset:relbert/lexical_relation_classification dataset:metaeval/linguisticprobing dataset:metaeval/crowdflower dataset:metaeval/ethics dataset:emo dataset:google_wellformed_query dataset:tweets_hate_speech_detection dataset:has_part dataset:wnut_17 dataset:ncbi_disease dataset:acronym_identification dataset:jnlpba dataset:species_800 dataset:SpeedOfMagic/ontonotes_english dataset:blog_authorship_corpus dataset:launch/open_question_type dataset:health_fact dataset:commonsense_qa dataset:mc_taco dataset:ade_corpus_v2 dataset:prajjwal1/discosense dataset:circa dataset:YaHi/EffectiveFeedbackStudentWriting dataset:Ericwang/promptSentiment dataset:Ericwang/promptNLI dataset:Ericwang/promptSpoke dataset:Ericwang/promptProficiency dataset:Ericwang/promptGrammar dataset:Ericwang/promptCoherence dataset:PiC/phrase_similarity dataset:copenlu/scientific-exaggeration-detection dataset:quarel dataset:mwong/fever-evidence-related dataset:numer_sense dataset:dynabench/dynasent dataset:raquiba/Sarcasm_News_Headline dataset:sem_eval_2010_task_8 dataset:demo-org/auditor_review dataset:medmcqa dataset:aqua_rat dataset:RuyuanWan/Dynasent_Disagreement dataset:RuyuanWan/Politeness_Disagreement dataset:RuyuanWan/SBIC_Disagreement dataset:RuyuanWan/SChem_Disagreement dataset:RuyuanWan/Dilemmas_Disagreement dataset:lucasmccabe/logiqa dataset:wiki_qa dataset:metaeval/cycic_classification dataset:metaeval/cycic_multiplechoice dataset:metaeval/sts-companion dataset:metaeval/commonsense_qa_2.0 dataset:metaeval/lingnli dataset:metaeval/monotonicity-entailment dataset:metaeval/arct dataset:metaeval/scinli dataset:metaeval/naturallogic dataset:onestop_qa dataset:demelin/moral_stories dataset:corypaik/prost dataset:aps/dynahate dataset:metaeval/syntactic-augmentation-nli dataset:metaeval/autotnli dataset:lasha-nlp/CONDAQA dataset:openai/webgpt_comparisons dataset:Dahoas/synthetic-instruct-gptj-pairwise dataset:metaeval/scruples dataset:metaeval/wouldyourather dataset:sileod/attempto-nli dataset:metaeval/defeasible-nli dataset:metaeval/help-nli dataset:metaeval/nli-veridicality-transitivity dataset:metaeval/natural-language-satisfiability dataset:metaeval/lonli dataset:metaeval/dadc-limit-nli dataset:ColumbiaNLP/FLUTE dataset:metaeval/strategy-qa dataset:openai/summarize_from_feedback dataset:metaeval/folio dataset:metaeval/tomi-nli dataset:metaeval/avicenna dataset:stanfordnlp/SHP dataset:GBaker/MedQA-USMLE-4-options-hf dataset:sileod/wikimedqa dataset:declare-lab/cicero dataset:amydeng2000/CREAK dataset:metaeval/mutual dataset:inverse-scaling/NeQA dataset:inverse-scaling/quote-repetition dataset:inverse-scaling/redefine-math dataset:metaeval/puzzte dataset:metaeval/implicatures dataset:race dataset:metaeval/spartqa-yn dataset:metaeval/spartqa-mchoice dataset:metaeval/temporal-nli arxiv:2301.05948 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
datasets:
- glue
- super_glue
- anli
- metaeval/babi_nli
- sick
- snli
- scitail
- hans
- alisawuffles/WANLI
- metaeval/recast
- sileod/probability_words_nli
- joey234/nan-nli
- pietrolesci/nli_fever
- pietrolesci/breaking_nli
- pietrolesci/conj_nli
- pietrolesci/fracas
- pietrolesci/dialogue_nli
- pietrolesci/mpe
- pietrolesci/dnc
- pietrolesci/gpt3_nli
- pietrolesci/recast_white
- pietrolesci/joci
- martn-nguyen/contrast_nli
- pietrolesci/robust_nli
- pietrolesci/robust_nli_is_sd
- pietrolesci/robust_nli_li_ts
- pietrolesci/gen_debiased_nli
- pietrolesci/add_one_rte
- metaeval/imppres
- pietrolesci/glue_diagnostics
- hlgd
- paws
- quora
- medical_questions_pairs
- conll2003
- Anthropic/hh-rlhf
- Anthropic/model-written-evals
- truthful_qa
- nightingal3/fig-qa
- tasksource/bigbench
- bigbench
- blimp
- cos_e
- cosmos_qa
- dream
- openbookqa
- qasc
- quartz
- quail
- head_qa
- sciq
- social_i_qa
- wiki_hop
- wiqa
- piqa
- hellaswag
- pkavumba/balanced-copa
- 12ml/e-CARE
- art
- tasksource/mmlu
- winogrande
- codah
- ai2_arc
- definite_pronoun_resolution
- swag
- math_qa
- metaeval/utilitarianism
- mteb/amazon_counterfactual
- SetFit/insincere-questions
- SetFit/toxic_conversations
- turingbench/TuringBench
- trec
- tals/vitaminc
- hope_edi
- strombergnlp/rumoureval_2019
- ethos
- tweet_eval
- discovery
- pragmeval
- silicone
- lex_glue
- papluca/language-identification
- imdb
- rotten_tomatoes
- ag_news
- yelp_review_full
- financial_phrasebank
- poem_sentiment
- dbpedia_14
- amazon_polarity
- app_reviews
- hate_speech18
- sms_spam
- humicroedit
- snips_built_in_intents
- banking77
- hate_speech_offensive
- yahoo_answers_topics
- pacovaldez/stackoverflow-questions
- zapsdcn/hyperpartisan_news
- zapsdcn/sciie
- zapsdcn/citation_intent
- go_emotions
- scicite
- liar
- relbert/lexical_relation_classification
- metaeval/linguisticprobing
- metaeval/crowdflower
- metaeval/ethics
- emo
- google_wellformed_query
- tweets_hate_speech_detection
- has_part
- wnut_17
- ncbi_disease
- acronym_identification
- jnlpba
- species_800
- SpeedOfMagic/ontonotes_english
- blog_authorship_corpus
- launch/open_question_type
- health_fact
- commonsense_qa
- mc_taco
- ade_corpus_v2
- prajjwal1/discosense
- circa
- YaHi/EffectiveFeedbackStudentWriting
- Ericwang/promptSentiment
- Ericwang/promptNLI
- Ericwang/promptSpoke
- Ericwang/promptProficiency
- Ericwang/promptGrammar
- Ericwang/promptCoherence
- PiC/phrase_similarity
- copenlu/scientific-exaggeration-detection
- quarel
- mwong/fever-evidence-related
- numer_sense
- dynabench/dynasent
- raquiba/Sarcasm_News_Headline
- sem_eval_2010_task_8
- demo-org/auditor_review
- medmcqa
- aqua_rat
- RuyuanWan/Dynasent_Disagreement
- RuyuanWan/Politeness_Disagreement
- RuyuanWan/SBIC_Disagreement
- RuyuanWan/SChem_Disagreement
- RuyuanWan/Dilemmas_Disagreement
- lucasmccabe/logiqa
- wiki_qa
- metaeval/cycic_classification
- metaeval/cycic_multiplechoice
- metaeval/sts-companion
- metaeval/commonsense_qa_2.0
- metaeval/lingnli
- metaeval/monotonicity-entailment
- metaeval/arct
- metaeval/scinli
- metaeval/naturallogic
- onestop_qa
- demelin/moral_stories
- corypaik/prost
- aps/dynahate
- metaeval/syntactic-augmentation-nli
- metaeval/autotnli
- lasha-nlp/CONDAQA
- openai/webgpt_comparisons
- Dahoas/synthetic-instruct-gptj-pairwise
- metaeval/scruples
- metaeval/wouldyourather
- sileod/attempto-nli
- metaeval/defeasible-nli
- metaeval/help-nli
- metaeval/nli-veridicality-transitivity
- metaeval/natural-language-satisfiability
- metaeval/lonli
- metaeval/dadc-limit-nli
- ColumbiaNLP/FLUTE
- metaeval/strategy-qa
- openai/summarize_from_feedback
- metaeval/folio
- metaeval/tomi-nli
- metaeval/avicenna
- stanfordnlp/SHP
- GBaker/MedQA-USMLE-4-options-hf
- sileod/wikimedqa
- declare-lab/cicero
- amydeng2000/CREAK
- metaeval/mutual
- inverse-scaling/NeQA
- inverse-scaling/quote-repetition
- inverse-scaling/redefine-math
- metaeval/puzzte
- metaeval/implicatures
- race
- metaeval/spartqa-yn
- metaeval/spartqa-mchoice
- metaeval/temporal-nli
language: en
library_name: transformers
license: apache-2.0
metrics:
- accuracy
pipeline_tag: zero-shot-classification
tags:
- deberta-v3-large
- text-classification
- nli
- natural-language-inference
- multitask
- multi-task
- pipeline
- extreme-multi-task
- extreme-mtl
- tasksource
- zero-shot
- rlhf
---

# Model Card for DeBERTa-v3-large-tasksource-nli

DeBERTa-v3-large fine-tuned with multi-task learning on 600 tasks of the [tasksource collection](https://github.com/sileod/tasksource/)
You can further fine-tune this model to use it for any classification or multiple-choice task.
This checkpoint has strong zero-shot validation performance on many tasks (e.g. 77% on WNLI).
The untuned model CLS embedding also has strong linear probing performance (90% on MNLI), due to the multitask training.

This is the shared model with the MNLI classifier on top. Its encoder was trained on many datasets including bigbench, Anthropic rlhf, anli... alongside many NLI and classification tasks with a SequenceClassification heads while using only one shared encoder.
Each task had a specific CLS embedding, which is dropped 10% of the time to facilitate model use without it. All multiple-choice model used the same classification layers. For classification tasks, models shared weights if their labels matched.
The number of examples per task was capped to 64k. The model was trained for 80k steps with a batch size of 384, and a peak learning rate of 2e-5.


tasksource training code: https://colab.research.google.com/drive/1iB4Oxl9_B5W3ZDzXoWJN-olUbqLBxgQS?usp=sharing

### Software
https://github.com/sileod/tasksource/ \
https://github.com/sileod/tasknet/ \
Training took 6 days on Nvidia A100 40GB GPU.



# Citation

More details on this [article:](https://arxiv.org/abs/2301.05948) 
```bib
@article{sileo2023tasksource,
  title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation},
  author={Sileo, Damien},
  url= {https://arxiv.org/abs/2301.05948},
  journal={arXiv preprint arXiv:2301.05948},
  year={2023}
}
```

# Loading a specific classifier
Classifiers for all tasks available. See https://huggingface.co/sileod/deberta-v3-large-tasksource-adapters

<img src="https://www.dropbox.com/s/eyfw8i1ekzxj3fa/task_embeddings.png?dl=1"  width="1000" height="">


# Model Card Contact

damien.sileo@inria.fr


</details>