README.md

metadata

license: apache-2.0
datasets: billsum
tags:
  - summarization
model-index:
  - name: d0r1h/LEDBill
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: billsum
          type: billsum
          config: default
          split: test
        metrics:
          - name: ROUGE-1
            type: rouge
            value: 38.6502
            verified: true
          - name: ROUGE-2
            type: rouge
            value: 18.5458
            verified: true
          - name: ROUGE-L
            type: rouge
            value: 25.6561
            verified: true
          - name: ROUGE-LSUM
            type: rouge
            value: 33.1575
            verified: true
          - name: loss
            type: loss
            value: 2.1305277347564697
            verified: true
          - name: gen_len
            type: gen_len
            value: 288.372
            verified: true

Longformer Encoder-Decoder (LED) fine-tuned on Billsum

This model is a fine-tuned version of led-base-16384 on the billsum dataset.

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, led-base-16384 was initialized from bart-base since both models share the exact same architecture. To be able to process 16K tokens, bart-base's position embedding matrix was simply copied 16 times.

How to use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)

case = "......."

input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
global_attention_mask = torch.zeros_like(input_ids)
global_attention_mask[:, 0] = 1

sequences = model.generate(input_ids, 
                           global_attention_mask=global_attention_mask).sequences
summary = tokenizer.batch_decode(sequences, 
                                 skip_special_tokens=True)
                                 

Evaluation results

When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:

Model	rouge1-f	rouge1-p	rouge2-f	rouge2-p	rougeL-f	rougeL-p
LEDBill	34	37	15	16	30	32
led-base	2	15	0	0	2	15

This notebook shows how led can effectively be used for downstream task such summarization.