readme: This model is developed with transformers v4.13 with minor patch in this [fork](https://github.com/vuiseng9/transformers/tree/pegasus-v4p13).
  
  # Setup
  ```bash
  git clone https://github.com/vuiseng9/transformers
  cd transformers
  git checkout pegasus-v4p13 && git reset --hard 41eeb07
  # installation, set summarization dependency
  # . . .
  ```
  
  # Train
  ```bash
  #!/usr/bin/env bash
  
  export CUDA_VISIBLE_DEVICES=0,1,2,3
  
  NEPOCH=10
  RUNID=pegasus-billsum-${NEPOCH}eph-run1
  OUTDIR=/data1/vchua/pegasus-hf4p13/pegasus/${RUNID}
  mkdir -p $OUTDIR
  
  nohup python run_summarization.py \
      --model_name_or_path google/pegasus-large \
      --dataset_name billsum \
      --do_train \
      --adafactor \
      --learning_rate 2e-4 \
      --label_smoothing_factor 0.1 \
      --num_train_epochs $NEPOCH \
      --per_device_train_batch_size 2 \
      --do_eval \
      --per_device_eval_batch_size 2 \
      --num_beams 8 \
      --max_source_length 1024 \
      --max_target_length 256 \
      --evaluation_strategy steps \
      --eval_steps 1000 \
      --save_strategy steps \
      --save_steps 2000 \
      --logging_steps 1 \
      --overwrite_output_dir \
      --run_name $RUNID \
      --output_dir $OUTDIR > $OUTDIR/run.log 2>&1 &
  ```
  
  # Eval
  ```bash
  #!/usr/bin/env bash
  
  export CUDA_VISIBLE_DEVICES=3
  
  DT=$(date +%F_%H-%M)
  RUNID=pegasus-billsum-${DT}
  OUTDIR=/data1/vchua/pegasus-hf4p13/pegasus-test/${RUNID}
  mkdir -p $OUTDIR
  
  nohup python run_summarization.py \
      --model_name_or_path vuiseng9/pegasus-billsum \
      --dataset_name billsum \
      --max_source_length 1024 \
      --max_target_length 256 \
      --do_predict \
      --per_device_eval_batch_size 8 \
      --predict_with_generate \
      --num_beams 8 \
      --overwrite_output_dir \
      --run_name $RUNID \
      --output_dir $OUTDIR > $OUTDIR/run.log 2>&1 &
  ```
  
  Although fine-tuning is carried out for 10 epochs, this model is the checkpoint (@12000 steps, 6.6epoch, 210mins) with lowest eval loss during training. Test/predict with this checkpoint should give results below.
  
  ```
  ***** predict metrics *****
    predict_gen_len            =   179.7363
    predict_loss               =     1.2452
    predict_rouge1             =    56.8657
    predict_rouge2             =    38.6531
    predict_rougeL             =    44.8399
    predict_rougeLsum          =    51.6266
    predict_runtime            = 1:19:28.20
    predict_samples            =       3269
    predict_samples_per_second =      0.686
    predict_steps_per_second   =      0.086
  ```