readme: ---
  language: en
  inference: false
  tags:
  - text-generation
  license: other
  commercial: false
  model-index:
  - name: inverse-scaling/opt-350m_eval
    results:
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: inverse-scaling/NeQA
        type: inverse-scaling/NeQA
        config: inverse-scaling--NeQA
        split: train
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.4666666666666667
        verified: true
      - name: Loss
        type: loss
        value: 0.9192380222864449
        verified: true
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: inverse-scaling/quote-repetition
        type: inverse-scaling/quote-repetition
        config: inverse-scaling--quote-repetition
        split: train
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.9633333333333334
        verified: true
      - name: Loss
        type: loss
        value: 0.03444786100047819
        verified: true
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: inverse-scaling/redefine-math
        type: inverse-scaling/redefine-math
        config: inverse-scaling--redefine-math
        split: train
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.6877777777777778
        verified: true
      - name: Loss
        type: loss
        value: 0.6016371671193176
        verified: true
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: inverse-scaling/hindsight-neglect-10shot
        type: inverse-scaling/hindsight-neglect-10shot
        config: inverse-scaling--hindsight-neglect-10shot
        split: train
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.4380952380952381
        verified: true
      - name: Loss
        type: loss
        value: 0.8774787804555325
        verified: true
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: mathemakitten/winobias_antistereotype_test_cot_v3
        type: mathemakitten/winobias_antistereotype_test_cot_v3
        config: mathemakitten--winobias_antistereotype_test_cot_v3
        split: test
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.44660194174757284
        verified: true
      - name: Loss
        type: loss
        value: 0.9301078982717057
        verified: true
    - task:
        type: zero-shot-classification
        name: Zero-Shot Text Classification
      dataset:
        name: mathemakitten/winobias_antistereotype_test_v5
        type: mathemakitten/winobias_antistereotype_test_v5
        config: mathemakitten--winobias_antistereotype_test_v5
        split: test
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.4368932038834951
        verified: true
      - name: Loss
        type: loss
        value: 0.9175132444057151
        verified: true
  ---
  
  
  # OPT : Open Pre-trained Transformer Language Models
  
  OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
  
  **Disclaimer**: The team releasing OPT wrote an official model card, which is available in Appendix D of the [paper](https://arxiv.org/pdf/2205.01068.pdf). 
  Content from **this** model card has been written by the Hugging Face team.
  
  ## Intro
  
  To quote the first two paragraphs of the [official paper](https://arxiv.org/abs/2205.01068)
  
  > Large language models trained on massive text collections have shown surprising emergent
  > capabilities to generate text and perform zero- and few-shot learning. While in some cases the public
  > can interact with these models through paid APIs, full model access is currently limited to only a
  > few highly resourced labs. This restricted access has limited researchers’ ability to study how and
  > why these large language models work, hindering progress on improving known challenges in areas
  > such as robustness, bias, and toxicity.
  
  > We present Open Pretrained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M
  > to 175B parameters, which we aim to fully and responsibly share with interested researchers. We train the OPT models to roughly match 
  > the performance and sizes of the GPT-3 class of models, while also applying the latest best practices in data
  > collection and efficient training. Our aim in developing this suite of OPT models is to enable reproducible and responsible research at scale, and
  > to bring more voices to the table in studying the impact of these LLMs. Definitions of risk, harm, bias, and toxicity, etc., should be articulated by the
  > collective research community as a whole, which is only possible when models are available for study.
  
  ## Model description
  
  OPT was predominantly pretrained with English text, but a small amount of non-English data is