readme: ---
  language: en
  license: apache-2.0
  datasets:
  - scientific_papers
  tags:
  - summarization
  model-index:
  - name: google/bigbird-pegasus-large-arxiv
    results:
    - task:
        type: summarization
        name: Summarization
      dataset:
        name: scientific_papers
        type: scientific_papers
        config: pubmed
        split: test
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 36.0276
        verified: true
      - name: ROUGE-2
        type: rouge
        value: 13.4166
        verified: true
      - name: ROUGE-L
        type: rouge
        value: 21.9612
        verified: true
      - name: ROUGE-LSUM
        type: rouge
        value: 29.648
        verified: true
      - name: loss
        type: loss
        value: 2.774355173110962
        verified: true
      - name: meteor
        type: meteor
        value: 0.2824
        verified: true
      - name: gen_len
        type: gen_len
        value: 209.2537
        verified: true
    - task:
        type: summarization
        name: Summarization
      dataset:
        name: cnn_dailymail
        type: cnn_dailymail
        config: 3.0.0
        split: test
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 9.0885
        verified: true
      - name: ROUGE-2
        type: rouge
        value: 1.0325
        verified: true
      - name: ROUGE-L
        type: rouge
        value: 7.3182
        verified: true
      - name: ROUGE-LSUM
        type: rouge
        value: 8.1455
        verified: true
      - name: loss
        type: loss
        value: .nan
        verified: true
      - name: gen_len
        type: gen_len
        value: 210.4762
        verified: true
    - task:
        type: summarization
        name: Summarization
      dataset:
        name: xsum
        type: xsum
        config: default
        split: test
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 4.9787
        verified: true
      - name: ROUGE-2
        type: rouge
        value: 0.3527
        verified: true
      - name: ROUGE-L
        type: rouge
        value: 4.3679
        verified: true
      - name: ROUGE-LSUM
        type: rouge
        value: 4.1723
        verified: true
      - name: loss
        type: loss
        value: .nan
        verified: true
      - name: gen_len
        type: gen_len
        value: 230.4886
        verified: true
    - task:
        type: summarization
        name: Summarization
      dataset:
        name: scientific_papers
        type: scientific_papers
        config: arxiv
        split: test
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 43.4702
        verified: true
      - name: ROUGE-2
        type: rouge
        value: 17.4297
        verified: true
      - name: ROUGE-L
        type: rouge
        value: 26.2587
        verified: true
      - name: ROUGE-LSUM
        type: rouge
        value: 35.5587
        verified: true
      - name: loss
        type: loss
        value: 2.1113228797912598
        verified: true
      - name: gen_len
        type: gen_len
        value: 183.3702
        verified: true
    - task:
        type: summarization
        name: Summarization
      dataset:
        name: samsum
        type: samsum
        config: samsum
        split: test
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 3.621
        verified: true
      - name: ROUGE-2
        type: rouge
        value: 0.1699
        verified: true
      - name: ROUGE-L
        type: rouge
        value: 3.2016
        verified: true
      - name: ROUGE-LSUM
        type: rouge
        value: 3.3269
        verified: true
      - name: loss
        type: loss
        value: 7.664482116699219
        verified: true
      - name: gen_len
        type: gen_len
        value: 233.8107
        verified: true
  ---
  
  # BigBirdPegasus model (large)
  
  BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle. 
  
  BigBird was introduced in this [paper](https://arxiv.org/abs/2007.14062) and first released in this [repository](https://github.com/google-research/bigbird).
  
  Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.
  
  ## Model description
  
  BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.
  
  ## How to use
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer
  
  tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")
  
  # by default encoder-attention is `block_sparse` with num_random_blocks=3, block_size=64
  model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv")
  
  # decoder attention type can't be cha