transformers transformer multimodal vqa text audio audio-text-to-text en zh dataset:zeroMN/nlp_corpus_zh dataset:zeroMN/hanlp_date-zh dataset:nyu-mll/glue dataset:aps/super_glue dataset:facebook/anli dataset:tasksource/babi_nli dataset:zeroMN/AVEdate dataset:sick dataset:snli dataset:scitail dataset:hans dataset:alisawuffles/WANLI dataset:tasksource/recast dataset:sileod/probability_words_nli dataset:joey234/nan-nli dataset:pietrolesci/nli_fever dataset:pietrolesci/breaking_nli dataset:pietrolesci/conj_nli dataset:pietrolesci/fracas dataset:pietrolesci/dialogue_nli dataset:pietrolesci/mpe dataset:pietrolesci/dnc dataset:pietrolesci/recast_white dataset:pietrolesci/joci dataset:pietrolesci/robust_nli dataset:pietrolesci/robust_nli_is_sd dataset:pietrolesci/robust_nli_li_ts dataset:pietrolesci/gen_debiased_nli dataset:pietrolesci/add_one_rte dataset:tasksource/imppres dataset:hlgd dataset:paws dataset:medical_questions_pairs dataset:Anthropic/model-written-evals dataset:truthful_qa dataset:nightingal3/fig-qa dataset:tasksource/bigbench dataset:blimp dataset:cos_e dataset:cosmos_qa dataset:dream dataset:openbookqa dataset:qasc dataset:quartz dataset:quail dataset:head_qa dataset:sciq dataset:social_i_qa dataset:wiki_hop dataset:wiqa dataset:piqa dataset:hellaswag dataset:pkavumba/balanced-copa dataset:12ml/e-CARE dataset:art dataset:winogrande dataset:codah dataset:ai2_arc dataset:definite_pronoun_resolution dataset:swag dataset:math_qa dataset:metaeval/utilitarianism dataset:mteb/amazon_counterfactual dataset:SetFit/insincere-questions dataset:SetFit/toxic_conversations dataset:turingbench/TuringBench dataset:trec dataset:tals/vitaminc dataset:hope_edi dataset:strombergnlp/rumoureval_2019 dataset:ethos dataset:tweet_eval dataset:discovery dataset:pragmeval dataset:silicone dataset:lex_glue dataset:papluca/language-identification dataset:imdb dataset:rotten_tomatoes dataset:ag_news dataset:yelp_review_full dataset:financial_phrasebank dataset:poem_sentiment dataset:dbpedia_14 dataset:amazon_polarity dataset:app_reviews dataset:hate_speech18 dataset:sms_spam dataset:humicroedit dataset:snips_built_in_intents dataset:hate_speech_offensive dataset:yahoo_answers_topics dataset:pacovaldez/stackoverflow-questions dataset:zapsdcn/hyperpartisan_news dataset:zapsdcn/sciie dataset:zapsdcn/citation_intent dataset:go_emotions dataset:allenai/scicite dataset:liar dataset:relbert/lexical_relation_classification dataset:tasksource/linguisticprobing dataset:tasksource/crowdflower dataset:metaeval/ethics dataset:emo dataset:google_wellformed_query dataset:tweets_hate_speech_detection dataset:has_part dataset:blog_authorship_corpus dataset:launch/open_question_type dataset:health_fact dataset:commonsense_qa dataset:mc_taco dataset:ade_corpus_v2 dataset:prajjwal1/discosense dataset:circa dataset:PiC/phrase_similarity dataset:copenlu/scientific-exaggeration-detection dataset:quarel dataset:mwong/fever-evidence-related dataset:numer_sense dataset:dynabench/dynasent dataset:raquiba/Sarcasm_News_Headline dataset:sem_eval_2010_task_8 dataset:demo-org/auditor_review dataset:medmcqa dataset:RuyuanWan/Dynasent_Disagreement dataset:RuyuanWan/Politeness_Disagreement dataset:RuyuanWan/SBIC_Disagreement dataset:RuyuanWan/SChem_Disagreement dataset:RuyuanWan/Dilemmas_Disagreement dataset:lucasmccabe/logiqa dataset:wiki_qa dataset:tasksource/cycic_classification dataset:tasksource/cycic_multiplechoice dataset:tasksource/sts-companion dataset:tasksource/commonsense_qa_2.0 dataset:tasksource/lingnli dataset:tasksource/monotonicity-entailment dataset:tasksource/arct dataset:tasksource/scinli dataset:tasksource/naturallogic dataset:onestop_qa dataset:demelin/moral_stories dataset:corypaik/prost dataset:aps/dynahate dataset:metaeval/syntactic-augmentation-nli dataset:tasksource/autotnli dataset:lasha-nlp/CONDAQA dataset:openai/webgpt_comparisons dataset:Dahoas/synthetic-instruct-gptj-pairwise dataset:metaeval/scruples dataset:metaeval/wouldyourather dataset:metaeval/defeasible-nli dataset:tasksource/help-nli dataset:metaeval/nli-veridicality-transitivity dataset:tasksource/lonli dataset:tasksource/dadc-limit-nli dataset:ColumbiaNLP/FLUTE dataset:tasksource/strategy-qa dataset:openai/summarize_from_feedback dataset:tasksource/folio dataset:yale-nlp/FOLIO dataset:tasksource/tomi-nli dataset:tasksource/avicenna dataset:stanfordnlp/SHP dataset:GBaker/MedQA-USMLE-4-options-hf dataset:sileod/wikimedqa dataset:declare-lab/cicero dataset:amydeng2000/CREAK dataset:tasksource/mutual dataset:inverse-scaling/NeQA dataset:inverse-scaling/quote-repetition dataset:inverse-scaling/redefine-math dataset:tasksource/puzzte dataset:tasksource/implicatures dataset:race dataset:tasksource/race-c dataset:tasksource/spartqa-yn dataset:tasksource/spartqa-mchoice dataset:tasksource/temporal-nli dataset:riddle_sense dataset:tasksource/clcd-english dataset:maximedb/twentyquestions dataset:metaeval/reclor dataset:tasksource/counterfactually-augmented-imdb dataset:tasksource/counterfactually-augmented-snli dataset:metaeval/cnli dataset:tasksource/boolq-natural-perturbations dataset:metaeval/acceptability-prediction dataset:metaeval/equate dataset:tasksource/ScienceQA_text_only dataset:Jiangjie/ekar_english dataset:tasksource/implicit-hate-stg1 dataset:metaeval/chaos-mnli-ambiguity dataset:IlyaGusev/headline_cause dataset:tasksource/logiqa-2.0-nli dataset:tasksource/oasst2_dense_flat dataset:sileod/mindgames dataset:metaeval/ambient dataset:metaeval/path-naturalness-prediction dataset:civil_comments dataset:AndyChiang/cloth dataset:AndyChiang/dgen dataset:tasksource/I2D2 dataset:webis/args_me dataset:webis/Touche23-ValueEval dataset:tasksource/starcon dataset:PolyAI/banking77 dataset:tasksource/ConTRoL-nli dataset:tasksource/tracie dataset:tasksource/sherliic dataset:tasksource/sen-making dataset:tasksource/winowhy dataset:tasksource/robustLR dataset:CLUTRR/v1 dataset:tasksource/logical-fallacy dataset:tasksource/parade dataset:tasksource/cladder dataset:tasksource/subjectivity dataset:tasksource/MOH dataset:tasksource/VUAC dataset:tasksource/TroFi dataset:sharc_modified dataset:tasksource/conceptrules_v2 dataset:metaeval/disrpt dataset:tasksource/zero-shot-label-nli dataset:tasksource/com2sense dataset:tasksource/scone dataset:tasksource/winodict dataset:tasksource/fool-me-twice dataset:tasksource/monli dataset:tasksource/corr2cause dataset:lighteval/lsat_qa dataset:tasksource/apt dataset:zeroshot/twitter-financial-news-sentiment dataset:tasksource/icl-symbol-tuning-instruct dataset:tasksource/SpaceNLI dataset:sihaochen/propsegment dataset:HannahRoseKirk/HatemojiBuild dataset:tasksource/regset dataset:tasksource/esci dataset:lmsys/chatbot_arena_conversations dataset:neurae/dnd_style_intents dataset:hitachi-nlp/FLD.v2 dataset:tasksource/SDOH-NLI dataset:allenai/scifact_entailment dataset:tasksource/feasibilityQA dataset:tasksource/simple_pair dataset:tasksource/AdjectiveScaleProbe-nli dataset:tasksource/resnli dataset:tasksource/SpaRTUN dataset:tasksource/ReSQ dataset:tasksource/semantic_fragments_nli dataset:MoritzLaurer/dataset_train_nli dataset:tasksource/stepgame dataset:tasksource/nlgraph dataset:tasksource/oasst2_pairwise_rlhf_reward dataset:tasksource/hh-rlhf dataset:tasksource/ruletaker dataset:qbao775/PARARULE-Plus dataset:tasksource/proofwriter dataset:tasksource/logical-entailment dataset:tasksource/nope dataset:tasksource/LogicNLI dataset:kiddothe2b/contract-nli dataset:AshtonIsNotHere/nli4ct_semeval2024 dataset:tasksource/lsat-ar dataset:tasksource/lsat-rc dataset:AshtonIsNotHere/biosift-nli dataset:tasksource/brainteasers dataset:Anthropic/persuasion dataset:erbacher/AmbigNQ-clarifying-question dataset:tasksource/SIGA-nli dataset:unigram/FOL-nli dataset:tasksource/goal-step-wikihow dataset:GGLab/PARADISE dataset:tasksource/doc-nli dataset:tasksource/mctest-nli dataset:tasksource/patent-phrase-similarity dataset:tasksource/natural-language-satisfiability dataset:tasksource/idioms-nli dataset:tasksource/lifecycle-entailment dataset:nvidia/HelpSteer dataset:nvidia/HelpSteer2 dataset:sadat2307/MSciNLI dataset:pushpdeep/UltraFeedback-paired dataset:tasksource/AES2-essay-scoring dataset:tasksource/english-grading dataset:tasksource/wice dataset:Dzeniks/hover dataset:sileod/missing-item-prediction dataset:tasksource/tasksource_dpo_pairs license:apache-2.0 model-index endpoints_compatible region:us
---
datasets:
- zeroMN/nlp_corpus_zh
- zeroMN/hanlp_date-zh
- nyu-mll/glue
- aps/super_glue
- facebook/anli
- tasksource/babi_nli
- zeroMN/AVEdate
- sick
- snli
- scitail
- hans
- alisawuffles/WANLI
- tasksource/recast
- sileod/probability_words_nli
- joey234/nan-nli
- pietrolesci/nli_fever
- pietrolesci/breaking_nli
- pietrolesci/conj_nli
- pietrolesci/fracas
- pietrolesci/dialogue_nli
- pietrolesci/mpe
- pietrolesci/dnc
- pietrolesci/recast_white
- pietrolesci/joci
- pietrolesci/robust_nli
- pietrolesci/robust_nli_is_sd
- pietrolesci/robust_nli_li_ts
- pietrolesci/gen_debiased_nli
- pietrolesci/add_one_rte
- tasksource/imppres
- hlgd
- paws
- medical_questions_pairs
- Anthropic/model-written-evals
- truthful_qa
- nightingal3/fig-qa
- tasksource/bigbench
- blimp
- cos_e
- cosmos_qa
- dream
- openbookqa
- qasc
- quartz
- quail
- head_qa
- sciq
- social_i_qa
- wiki_hop
- wiqa
- piqa
- hellaswag
- pkavumba/balanced-copa
- 12ml/e-CARE
- art
- winogrande
- codah
- ai2_arc
- definite_pronoun_resolution
- swag
- math_qa
- metaeval/utilitarianism
- mteb/amazon_counterfactual
- SetFit/insincere-questions
- SetFit/toxic_conversations
- turingbench/TuringBench
- trec
- tals/vitaminc
- hope_edi
- strombergnlp/rumoureval_2019
- ethos
- tweet_eval
- discovery
- pragmeval
- silicone
- lex_glue
- papluca/language-identification
- imdb
- rotten_tomatoes
- ag_news
- yelp_review_full
- financial_phrasebank
- poem_sentiment
- dbpedia_14
- amazon_polarity
- app_reviews
- hate_speech18
- sms_spam
- humicroedit
- snips_built_in_intents
- hate_speech_offensive
- yahoo_answers_topics
- pacovaldez/stackoverflow-questions
- zapsdcn/hyperpartisan_news
- zapsdcn/sciie
- zapsdcn/citation_intent
- go_emotions
- allenai/scicite
- liar
- relbert/lexical_relation_classification
- tasksource/linguisticprobing
- tasksource/crowdflower
- metaeval/ethics
- emo
- google_wellformed_query
- tweets_hate_speech_detection
- has_part
- blog_authorship_corpus
- launch/open_question_type
- health_fact
- commonsense_qa
- mc_taco
- ade_corpus_v2
- prajjwal1/discosense
- circa
- PiC/phrase_similarity
- copenlu/scientific-exaggeration-detection
- quarel
- mwong/fever-evidence-related
- numer_sense
- dynabench/dynasent
- raquiba/Sarcasm_News_Headline
- sem_eval_2010_task_8
- demo-org/auditor_review
- medmcqa
- RuyuanWan/Dynasent_Disagreement
- RuyuanWan/Politeness_Disagreement
- RuyuanWan/SBIC_Disagreement
- RuyuanWan/SChem_Disagreement
- RuyuanWan/Dilemmas_Disagreement
- lucasmccabe/logiqa
- wiki_qa
- tasksource/cycic_classification
- tasksource/cycic_multiplechoice
- tasksource/sts-companion
- tasksource/commonsense_qa_2.0
- tasksource/lingnli
- tasksource/monotonicity-entailment
- tasksource/arct
- tasksource/scinli
- tasksource/naturallogic
- onestop_qa
- demelin/moral_stories
- corypaik/prost
- aps/dynahate
- metaeval/syntactic-augmentation-nli
- tasksource/autotnli
- lasha-nlp/CONDAQA
- openai/webgpt_comparisons
- Dahoas/synthetic-instruct-gptj-pairwise
- metaeval/scruples
- metaeval/wouldyourather
- metaeval/defeasible-nli
- tasksource/help-nli
- metaeval/nli-veridicality-transitivity
- tasksource/lonli
- tasksource/dadc-limit-nli
- ColumbiaNLP/FLUTE
- tasksource/strategy-qa
- openai/summarize_from_feedback
- tasksource/folio
- yale-nlp/FOLIO
- tasksource/tomi-nli
- tasksource/avicenna
- stanfordnlp/SHP
- GBaker/MedQA-USMLE-4-options-hf
- sileod/wikimedqa
- declare-lab/cicero
- amydeng2000/CREAK
- tasksource/mutual
- inverse-scaling/NeQA
- inverse-scaling/quote-repetition
- inverse-scaling/redefine-math
- tasksource/puzzte
- tasksource/implicatures
- race
- tasksource/race-c
- tasksource/spartqa-yn
- tasksource/spartqa-mchoice
- tasksource/temporal-nli
- riddle_sense
- tasksource/clcd-english
- maximedb/twentyquestions
- metaeval/reclor
- tasksource/counterfactually-augmented-imdb
- tasksource/counterfactually-augmented-snli
- metaeval/cnli
- tasksource/boolq-natural-perturbations
- metaeval/acceptability-prediction
- metaeval/equate
- tasksource/ScienceQA_text_only
- Jiangjie/ekar_english
- tasksource/implicit-hate-stg1
- metaeval/chaos-mnli-ambiguity
- IlyaGusev/headline_cause
- tasksource/logiqa-2.0-nli
- tasksource/oasst2_dense_flat
- sileod/mindgames
- metaeval/ambient
- metaeval/path-naturalness-prediction
- civil_comments
- AndyChiang/cloth
- AndyChiang/dgen
- tasksource/I2D2
- webis/args_me
- webis/Touche23-ValueEval
- tasksource/starcon
- PolyAI/banking77
- tasksource/ConTRoL-nli
- tasksource/tracie
- tasksource/sherliic
- tasksource/sen-making
- tasksource/winowhy
- tasksource/robustLR
- CLUTRR/v1
- tasksource/logical-fallacy
- tasksource/parade
- tasksource/cladder
- tasksource/subjectivity
- tasksource/MOH
- tasksource/VUAC
- tasksource/TroFi
- sharc_modified
- tasksource/conceptrules_v2
- metaeval/disrpt
- tasksource/zero-shot-label-nli
- tasksource/com2sense
- tasksource/scone
- tasksource/winodict
- tasksource/fool-me-twice
- tasksource/monli
- tasksource/corr2cause
- lighteval/lsat_qa
- tasksource/apt
- zeroshot/twitter-financial-news-sentiment
- tasksource/icl-symbol-tuning-instruct
- tasksource/SpaceNLI
- sihaochen/propsegment
- HannahRoseKirk/HatemojiBuild
- tasksource/regset
- tasksource/esci
- lmsys/chatbot_arena_conversations
- neurae/dnd_style_intents
- hitachi-nlp/FLD.v2
- tasksource/SDOH-NLI
- allenai/scifact_entailment
- tasksource/feasibilityQA
- tasksource/simple_pair
- tasksource/AdjectiveScaleProbe-nli
- tasksource/resnli
- tasksource/SpaRTUN
- tasksource/ReSQ
- tasksource/semantic_fragments_nli
- MoritzLaurer/dataset_train_nli
- tasksource/stepgame
- tasksource/nlgraph
- tasksource/oasst2_pairwise_rlhf_reward
- tasksource/hh-rlhf
- tasksource/ruletaker
- qbao775/PARARULE-Plus
- tasksource/proofwriter
- tasksource/logical-entailment
- tasksource/nope
- tasksource/LogicNLI
- kiddothe2b/contract-nli
- AshtonIsNotHere/nli4ct_semeval2024
- tasksource/lsat-ar
- tasksource/lsat-rc
- AshtonIsNotHere/biosift-nli
- tasksource/brainteasers
- Anthropic/persuasion
- erbacher/AmbigNQ-clarifying-question
- tasksource/SIGA-nli
- unigram/FOL-nli
- tasksource/goal-step-wikihow
- GGLab/PARADISE
- tasksource/doc-nli
- tasksource/mctest-nli
- tasksource/patent-phrase-similarity
- tasksource/natural-language-satisfiability
- tasksource/idioms-nli
- tasksource/lifecycle-entailment
- nvidia/HelpSteer
- nvidia/HelpSteer2
- sadat2307/MSciNLI
- pushpdeep/UltraFeedback-paired
- tasksource/AES2-essay-scoring
- tasksource/english-grading
- tasksource/wice
- Dzeniks/hover
- sileod/missing-item-prediction
- tasksource/tasksource_dpo_pairs
language:
- en
- zh
library_name: transformers
license: apache-2.0
metrics:
- accuracy
- bleu
- wer
pipeline_tag: audio-text-to-text
tags:
- multimodal
- vqa
- text
- audio
widget:
- text: My name is Sylvain and I live in Paris
  example_title: Parisian
- text: My name is Sarah and I live in London
  example_title: Londoner
model-index:
- name: Evolutionary Multi-Modal Model
  results:
  - task:
      type: vqa
      name: Visual Question Answering
    dataset:
      name: Synthetic Multimodal Dataset
      type: synthetic-dataset
      split: test
    metrics:
    - type: accuracy
      value: 85
---
### Model Sources
You need to use separate code, audio, text, and natural language together with the model. Because the model will use separate word segmenters and vocabularies to achieve the best results when dealing with special cases.
--

- **Repository:** [https://zeromn-zeromn-shmt.hf.space]
- **kaggle:** [https://www.kaggle.com/models/zeroeva/evolutionary-multi-modal) (https://www.kaggle.com/models/zeroeva/evolutionary-multi-modal)
- **Demo:** [https://zeromn-zeromn-shmt.hf.space]

##     Multi-Modal Model
# Model Card for Evolutionary

--
<script
	type="module"
	src="https://gradio.s3-us-west-2.amazonaws.com/5.12.0/gradio.js"
></script>

<gradio-app src="https://zeromn-zeromn-shmt.hf.space"></gradio-app>

-
### Model breast_cancer_wisconsin_original test

```python
from ucimlrepo import fetch_ucirepo 
fetch dataset 
breast_cancer_wisconsin_original = fetch_ucirepo(id=15) 
  
data (as pandas dataframes) 
X = breast_cancer_wisconsin_original.data.features 
y = breast_cancer_wisconsin_original.data.targets 
 
metadata 
print(breast_cancer_wisconsin_original.metadata) 
 
variable information 
print(breast_cancer_wisconsin_original.variables) 
```
##########################################################
-
#       0       0.93      0.99      0.96        79
#       1       0.98      0.90      0.94        58
--
#accuracy                           0.95       137
--
--
This model, named `Evolutionary Multi-Modal Model`, is a multimodal transformer designed to handle a variety of tasks including vision and audio processing. It is built on top of the `adapter-transformers` and `transformers` libraries and is intended to be a versatile base model for both direct use and fine-tuning.
-

--
**Developed by:** Independent researcher
**Funded by :** Self-funded
**Shared by :** Independent researcher
**Model type:** Multimodal
**Language(s) (NLP):** English zh
**License:** Apache-2.0
**Finetuned from model :** None
-

## Uses：https://huggingface.co/zeroMN/SHMT

### Direct Use
```python
git lfs install

git clone https://huggingface.co/zeroMN/SHMT.git
```
### Downstream Use

The model can be fine-tuned for specific tasks such as visual question answering (VQA), image captioning, and audio recognition. 

### Out-of-Scope Use

The Evolved Multimodal Model is not suitable for tasks that require high expertise or domain-specific expertise beyond its current capabilities. The number of speech frames still needs to be fine-tuned by yourself.
## Bias, Risks, and Limitations

### Recommendations

Users (both direct and downstream) should be made aware of the following risks, biases, and limitations:

- **Bias:** The model may exhibit biases present in the training data, particularly if the data is not representative of all populations.
- **Risks:** The model should not be used in critical applications where high accuracy and reliability are required without thorough testing and validation.
- **Limitations:** The model may not perform well on tasks that require fine-grained recognition or highly specialized audio processing.

## How to Get Started with the Model
```python
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zeroMN/SHMT")
```
```python
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("zeroMN/SHMT")
```