README.md

---
license: apache-2.0
tags:
- summarization
datasets: billsum
widget:
- text: 'The people of the State of California do enact as follows: SECTIONHEADER
    Section 1170.02 is added to the Penal Code, to read: 1170.02. A prisoner is not
    eligible for resentence or recall pursuant to subdivision (e) of Section 1170
    if he or she was convicted of first-degree murder if the victim was a peace officer,
    as defined in Section 830.1, 830.2, 830.3, 830.31, 830.32, 830.33, 830.34, 830.35,
    830.36, 830.37, 830.4, 830.5, 830.6, 830.10, 830.11, or 830.12, who was killed
    while engaged in the performance of his or her duties, and the individual knew,
    or reasonably should have known, that the victim was a peace officer engaged in
    the performance of his or her duties, or the victim was a peace officer or a former
    peace officer under any of the above-enumerated sections, and was intentionally
    killed in retaliation for the performance of his or her official duties. SECTIONHEADER
    Section 3550 of the Penal Code is amended to read: 3550. Notwithstanding any other
    law, except as provided in subdivision (b), if the head physician of an institution
    in which a prisoner is incarcerated determines, as provided in this section, that
    the prisoner is permanently medically incapacitated with a medical condition that
    renders him or her permanently unable to perform activities of basic daily living,
    and results in the prisoner requiring 24-hour care, and that incapacitation did
    not exist at the time of sentencing, the prisoner shall be granted medical parole
    if the Board of Parole Hearings determines that the conditions under which he
    or she would be released would not reasonably pose a threat to public safety.
    This section does not alter or diminish the rights conferred under the Victims
    Bill of Rights Act of 2008 . Subdivision (a) does not apply to any of the following:
    A prisoner sentenced to death or life in prison without possibility of parole.
    A prisoner who is serving a sentence for which parole, pursuant to subdivision
    (a), is prohibited by any initiative statute. A prisoner who was convicted of
    first-degree murder if the victim was a peace officer, as defined in Section 830.1,
    830.2, 830.3, 830.31, 830.32, 830.33, 830.34, 830.35, 830.36, 830.37, 830.4, 830.5,
    830.6, 830.10, 830.11, or 830.12, who was killed while engaged in the performance
    of his or her duties, and the individual knew, or reasonably should have known,
    that the victim was a peace officer engaged in the performance of his or her duties,
    or the victim was a peace officer or a former peace officer under any of the above-enumerated
    sections, and was intentionally killed in retaliation for the performance of his
    or her official duties. When a physician employed by the Department of Corrections
    and Rehabilitation who is the primary care provider for a prisoner identifies
    a prisoner that he or she believes meets the medical criteria for medical parole
    specified in subdivision (a), the primary care physician shall recommend to the
    head physician of the institution where the prisoner is located that the prisoner
    be referred to the Board of Parole Hearings for consideration for medical parole.
    Within 30 days of receiving that recommendation, if the head physician of the
    institution concurs in the recommendation of the primary care physician, he or
    she shall refer the matter to the Board of Parole Hearings using a standardized
    form and format developed by the department, and if the head physician of the
    institution does not concur in the recommendation, he or she shall provide the
    primary care physician with a written explanation of the reasons for denying the
    referral. Notwithstanding any other provisions of this section, the prisoner or
    his or her family member or designee may independently request consideration for
    medical parole by contacting the head physician at the prison or the department.
    Within 30 days of receiving the request, the head physician of the institution
    shall, in consultation with the prisoners primary care physician, make a determination
    regarding whether the prisoner meets the criteria for medical parole as specified
    in subdivision (a) and, if the head physician of the institution determines that
    the prisoner satisfies the criteria set forth in subdivision (a), he or she shall
    refer the matter to the Board of Parole Hearings using a standardized form and
    format developed by the department. If the head physician of the institution does
    not concur in the recommendation, he or she shall provide the prisoner or his
    or her family member or designee with a written explanation of the reasons for
    denying the application. The Department of Corrections and Rehabilitation shall
    complete parole plans for inmates referred to the Board of Parole Hearings for
    medical parole consideration. The parole plans shall include, but not be limited
    to, the inmates plan for residency and medical care. Notwithstanding any other
    law, medical parole hearings shall be conducted by two-person panels consisting
    of at least one commissioner. In the event of a tie vote, the matter shall be
    referred to the full board for a decision. Medical parole hearings may be heard
    in absentia. Upon receiving a recommendation from the head physician of the institution
    where a prisoner is located for the prisoner to be granted medical parole pursuant
    to subdivision (c) or (d), the board, as specified in subdivision (f), shall make
    an independent judgment regarding whether the conditions under which the inmate
    would be released pose a reasonable threat to public safety, and make written
    findings related thereto. Notwithstanding any other law, the board or the Division
    of Adult Parole Operations shall have the authority to impose any reasonable conditions
    on prisoners subject to medical parole supervision pursuant to subdivision (a),
    including, but not limited to, the requirement that the parolee submit to electronic
    monitoring. As a further condition of medical parole, pursuant to subdivision
    (a), the parolee may be required to submit to an examination by a physician selected
    by the board for the purpose of diagnosing the parolees current medical condition.
    In the event such an examination takes place, a report of the examination and
    diagnosis shall be submitted to the board by the examining physician. If the board
    determines, based on that medical examination, that the persons medical condition
    has improved to the extent that the person no longer qualifies for medical parole,
    the board shall return the person to the custody of the department. Notwithstanding
    any other law establishing maximum periods for parole, a prisoner sentenced to
    a determinate term who is placed on medical parole supervision prior to the earliest
    possible release date and who remains eligible for medical parole, shall remain
    on medical parole, pursuant to subdivision (a), until that earliest possible release
    date, at which time the parolee shall commence serving that period of parole provided
    by, and under the provisions of, Chapter 8 of Title 1. Notwithstanding any other
    law establishing maximum periods for parole, a prisoner sentenced to an indeterminate
    term who is placed on medical parole supervision prior to the prisoners minimum
    eligible parole date, and who remains eligible for medical parole, shall remain
    on medical parole pursuant to subdivision (a) until that minimum eligible parole
    date, at which time the parolee shall be eligible for parole consideration under
    all other provisions of Chapter 8 of Title 1. The Department of Corrections and
    Rehabilitation shall, at the time a prisoner is placed on medical parole supervision
    pursuant to subdivision (a), ensure that the prisoner has applied for any federal
    entitlement programs for which the prisoner is eligible, and has in his or her
    possession a discharge medical summary, full medical records, parole medications,
    and all property belonging to the prisoner that was under the control of the department.
    Any additional records shall be sent to the prisoners forwarding address after
    release to health care-related parole supervision. The provisions for medical
    parole set forth in this title shall not affect an inmates eligibility for any
    other form of parole or release provided by law. (1) Notwithstanding any other
    law, the Department of Corrections and Rehabilitation shall give notice to the
    county of commitment and the proposed county of release, if that county is different
    than the county of commitment, of any medical parole hearing as described in subdivision
    (f), and of any medical parole release as described in subdivision (g). Notice
    shall be made at least 30 days, or as soon as feasible, prior to the time any
    medical parole hearing or medical parole release is scheduled for an inmate receiving
    medical parole consideration, regardless of whether the inmate is sentenced either
    determinately or indeterminately.'
model-index:
- name: d0r1h/LEDBill
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: billsum
      type: billsum
      config: default
      split: test
    metrics:
    - type: rouge
      value: 38.6502
      name: ROUGE-1
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOWM2ZjlkN2E5YTU0Yjk3YWRmYTBkYzY2MDhhNGVjYTQ4MGJmY2ZlOGIyYzlmMDUxZjE0M2NkM2VjYjMxOWU1MiIsInZlcnNpb24iOjF9.fw-KAiHzPfdD7UGPs0PrELi_5RR1NpLlzfEn6vH4UpW5DELF2LqetsnbRMwyri2XkPEwtR_RhJySHPSjEcePBg
    - type: rouge
      value: 18.5458
      name: ROUGE-2
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMThmMTJmNGE5ZDc0ZmUwM2U0ZDk3Yjg5Zjg4MzJjMDEzMDczZTBhZDFiYzU1NzNmM2IzOWYwN2Y2YTE2MmY3ZiIsInZlcnNpb24iOjF9.6g9_iUWrxGkMFt1Jb_S0bibSLMVn3X34R8aFx680Uy2a11pwsy9E7t8mVYL3ygYWyO9ZTcutxR7rsq8zFtfoBg
    - type: rouge
      value: 25.6561
      name: ROUGE-L
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjc1M2QwMWE4NmExMDc1MDQ0ZmIwNDM5MDMyNDljNzNjZTMxZWY4NGExODFkZTkyNTU5ODgyNTRhZjhmZjdlNiIsInZlcnNpb24iOjF9.U7gn6EAclLOhRFHgKuMjwuDP1IEYi3ZFH1s3FNK4IIBE0hlxzHtbpLUYhkkA5jeB1MJ0zK2TfWR9f_OdZfORBw
    - type: rouge
      value: 33.1575
      name: ROUGE-LSUM
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDQ0ZmQzYzg3NWRkZTE5MzI2MTY0MmExYWM4YWI0ZDdhZDU5MmM2ZjZmNmUyYzQ0ZTY3MmM3NTgyODk3YmRmOSIsInZlcnNpb24iOjF9.DHNZbJcBc2XtX8OyXhmJ4vHveYcOcNUPMNK6wWtW4i3xlU15fN7uXXkiRtLWyk_Ka4niewl8XII9_EYTnE0HCA
    - type: loss
      value: 2.1305277347564697
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTZlOTY2NGNmZDI5OTY1ZTdkOWFjYmUwNTM3NzY3NGVlZTY3OTk2YjZjYjUwMjMxMjVjMWI3MGUwMjkwYTg5MiIsInZlcnNpb24iOjF9.6nd0t-nLY9mtD6THuuNoew0GslIji28ff_OJzAahNWBF4gYrpo68VRHFHiN2U47N9Y-ddcD7t9lzU8a-4ji_Cw
    - type: gen_len
      value: 288.372
      name: gen_len
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjFkOTY4NTcwYmQ0MjhmYzg0ZGM0MjU4Y2JlZTAwMjYzNTExMzE5YTk2YTVjOGIzZmM2OTZmMTQzODZhYjMxZCIsInZlcnNpb24iOjF9.6uJkenDAaem41bsCRtXe_ISO2ij1bZwkUv2m19kvqKx1xXpDrDO6F4n6v2z3PpwzCzncjxPTjS2w3A4wFl-UDw
---

# Longformer Encoder-Decoder (LED) fine-tuned on Billsum



This model is a fine-tuned version of [led-base-16384](https://huggingface.co/allenai/led-base-16384) on the [billsum](https://huggingface.co/datasets/billsum) dataset.

As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.


## How to use

```Python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)

case = "......."

input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
global_attention_mask = torch.zeros_like(input_ids)
global_attention_mask[:, 0] = 1

sequences = model.generate(input_ids, 
                           global_attention_mask=global_attention_mask).sequences
summary = tokenizer.batch_decode(sequences, 
                                 skip_special_tokens=True)
                                 
```


## Evaluation results

When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:

| Model |            rouge1-f  |  rouge1-p  |  rouge2-f   |  rouge2-p  | rougeL-f   |  rougeL-p  |
|:-----------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|
| LEDBill           | **34** |  **37** | **15**  | **16** | **30**  | **32** | 
| led-base          | 2     |  15     | 0      | 0     | 2      | 15 |

[This notebook](https://colab.research.google.com/drive/1iEEFbWeTGUSDesmxHIU2QDsPQM85Ka1K?usp=sharing) shows how *led* can effectively be used for downstream task such summarization.