readme: ---
  widget:
  - context: While deep and large pre-trained models are the state-of-the-art for various
      natural language processing tasks, their huge size poses significant challenges
      for practical uses in resource constrained settings. Recent works in knowledge
      distillation propose task-agnostic as well as task-specific methods to compress
      these models, with task-specific ones often yielding higher compression rate.
      In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
      that leverages the advantage of task-specific methods for learning a small universal
      model that can be applied to arbitrary tasks and languages. To this end, we study
      the transferability of several source tasks, augmentation resources and model
      architecture for distillation. We evaluate our model performance on multiple tasks,
      including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
      question answering dataset and a massive multi-lingual NER dataset with 41 languages.
    example_title: xtremedistil q1
    text: What is XtremeDistil?
  - context: While deep and large pre-trained models are the state-of-the-art for various
      natural language processing tasks, their huge size poses significant challenges
      for practical uses in resource constrained settings. Recent works in knowledge
      distillation propose task-agnostic as well as task-specific methods to compress
      these models, with task-specific ones often yielding higher compression rate.
      In this work, we develop a new task-agnostic distillation framework XtremeDistilTransformers
      that leverages the advantage of task-specific methods for learning a small universal
      model that can be applied to arbitrary tasks and languages. To this end, we study
      the transferability of several source tasks, augmentation resources and model
      architecture for distillation. We evaluate our model performance on multiple tasks,
      including the General Language Understanding Evaluation (GLUE) benchmark, SQuAD
      question answering dataset and a massive multi-lingual NER dataset with 41 languages.
    example_title: xtremedistil q2
    text: On what is the model validated?
  datasets:
  - squad_v2
  metrics:
  - f1
  - exact
  tags:
  - question-answering
  model-index:
  - name: nbroad/xdistil-l12-h384-squad2
    results:
    - task:
        type: question-answering
        name: Question Answering
      dataset:
        name: squad_v2
        type: squad_v2
        config: squad_v2
        split: validation
      metrics:
      - name: Exact Match
        type: exact_match
        value: 75.4591
        verified: true
      - name: F1
        type: f1
        value: 79.3321
        verified: true
    - task:
        type: question-answering
        name: Question Answering
      dataset:
        name: squad
        type: squad
        config: plain_text
        split: validation
      metrics:
      - name: Exact Match
        type: exact_match
        value: 81.8604
        verified: true
      - name: F1
        type: f1
        value: 89.6654
        verified: true
  ---
  
  xtremedistil-l12-h384 trained on SQuAD 2.0
  
  "eval_exact": 75.45691906005221  
  "eval_f1": 79.32502968532793