readme: ---
  language: en
  tags:
   - biomedical
   - text summarization
  datasets:
   - ccdv/pubmed-summarization
    
  license: apache-2.0
  ---
  
  ## Introduction
  [Google's LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/pdf/2112.07916.pdf). 
  
  This is an unofficial longt5-large-16384-pubmed-10k_steps checkpoint. I.e., this is a large configuration of the LongT5 model with a transient-global attention fine-tuned on [pubmed summarization dataset](https://huggingface.co/datasets/ccdv/pubmed-summarization) for 10,000 training steps. 
  
  ## Results and Fine-tuning Details 
  
  The fine-tuned model achieves the following results on the evaluation set using `beam_search=1` and without any specific calibration of generation parameters are presented below, altogether with the results from the original paper (the original scores are higher, very likely due to a higher number of training steps).
  
  | Metric | Score | Score (original paper)
  | --- | --- | --- |
  | Rouge-1 | 49.11 | 49.98 |
  | Rouge-2 | 23.66 | 24.69 |
  | Rouge-L | 31.19 | x |
  | Rouge-Lsum | 45.87 | 46.46 |
  
  Following previous [setup](https://huggingface.co/Stancld/longt5-tglobal-large-16384-pubmed-3k_steps/), the training parameters follow the ones specified in the paper. We accumulated batch size to 128 examples and used `Adafactor` optimizer with a constant learning rate `0.001`. The training took about 14 days on 2 A100 GPUs.
  
  The only specific adjustment, I made for the training, was to pad very short input articles (less than 16 tokens) to at least 16 tokens as this sequences do not contribute to gradient creation in the *transient-global* attention, which resulted in training crashes when DDP used.
  
  ## Usage
  
  ```python 
  LONG_ARTICLE = """"anxiety affects quality of life in those living
  with parkinson 's disease ( pd ) more so than
  overall cognitive status , motor deficits , apathy
  , and depression [ 13 ] . although anxiety and
  depression are often related and coexist in pd
  patients , recent research suggests that anxiety
  rather than depression is the most prominent and
  prevalent mood disorder in pd [ 5 , 6 ] . yet ,
  our current understanding of anxiety and its
  impact on cognition in pd , as well as its neural
  basis and best treatment practices , remains
  meager and lags far behind that of depression .
  overall , neuropsychiatric symptoms in pd have
  been shown to be negatively associated with
  cognitive performance . for example , higher
  depression scores have been correlated with lower
  scores on the mini - mental state exam ( mmse ) [
  8 , 9 ] as well as tests of memory and executive
  functions ( e.g. , attention ) [ 1014 ] . likewise
  , apathy and anhedonia in pd patients have been
  associated with executive dysfunction [ 10 , 1523
  ] . however , few studies have specifically
  investigated the relationship between anxiety and
  cognition in pd . one study showed a strong
  negative relationship between anxiety ( both state
  and trait ) and overall cognitive performance (
  measured by the total of the repeatable battery
  for the assessment of neuropsychological status
  index ) within a sample of 27 pd patients .
  furthermore , trait anxiety was negatively
  associated with each of the cognitive domains
  assessed by the rbans ( i.e. , immediate memory ,
  visuospatial construction , language , attention ,
  and delayed memory ) . two further studies have
  examined whether anxiety differentially affects
  cognition in patients with left - sided dominant
  pd ( lpd ) versus right - sided dominant pd ( rpd
  ) ; however , their findings were inconsistent .
  the first study found that working memory
  performance was worse in lpd patients with anxiety
  compared to rpd patients with anxiety , whereas
  the second study reported that , in lpd , apathy
  but not anxiety was associated with performance on
  nonverbally mediated executive functions and
  visuospatial tasks ( e.g. , tmt - b , wms - iii
  spatial span ) , while in rpd , anxiety but not
  apathy significantly correlated with performance
  on verbally mediated tasks ( e.g. , clock reading
  test and boston naming test ) . furthermore ,
  anxiety was significantly correlated with
  neuropsychological measures of attention and
  executive and visuospatial functions . taken
  together , it is evident that there are limited
  and inconsistent findings describing the
  relationship between anxiety and cognition in pd
  and more specifically how anxiety might influence
  particular domains of cognition such as attention
  and memory and executive functioning . it is also
  striking that , to date , no study has examined
  the influence of anxiety on cognition in pd by
  directly comparing groups of pd patients with and
  without anxiety while excluding depression . given
  that research on healthy young adults suggests
  that anxiety reduces processing capacity and
  impairs processing efficiency , especially in the
  central executive and attentional systems of
  working memory [ 26 , 27 ] , we hypothesized that
  pd patients with anxiety would show impairments in
  attentional set - shi