pegasus/cnn_dm  12-2 distillation performing poorly
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-1028-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help
 
@sshleifer 
## Information

I am trying to distil the pegasus model to reduce the runtime and memory requirements. I am following **No Teacher Distillation** approach. However, the model generates bad quality text.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): CNN
* [ ] my own task or dataset: (give details below)

## To reproduce
I have trained the model using below command:

**Download data:**
wget https://cdn-datasets.huggingface.co/summarization/cnn_dm_v2.tgz
tar -xzvf cnn_dm_v2.tgz  # empty lines removed
mv cnn_cln cnn_dm

**Command to train:**
python finetune.py --learning_rate=3e-5 --do_train --do_predict --val_check_interval 0.25 --n_val 500 --num_train_epochs 6 --freeze_encoder --freeze_embeds --data_dir ./cnn_dm/ --max_target_length 142 --val_max_target_length=142 --train_batch_size=1 --eval_batch_size=1 --gradient_accumulation_steps=256 --model_name_or_path sshleifer/student_pegasus_cnn_12_2 --tokenizer_name google/pegasus-cnn_dailymail --warmup_steps 500 --output_dir distilpegasus-cnn-12-2 --gpus 1 --adafactor --num_workers=0 --fp16_opt_level=O1 --fp16

**Inference code:**

```
from transformers import PegasusForConditionalGeneration, PegasusTokenizer, PegasusConfig
import torch

PEGASUS_MODEL = '/home/ubuntu/finetune/transformers/examples/seq2seq/distilpegasus-cnn-12-2/best_tfmr'
PEGASUS_TOKENIZER = 'google/pegasus-cnn_dailymail'

class PegasusSummarizer:
    def __init__(self):
        self.torch_device = 'cpu'
        self.tokenizer = PegasusTokenizer.from_pretrained(PEGASUS_TOKENIZER)
        self.model = PegasusForConditionalGeneration.from_pretrained(PEGASUS_MODEL).to(self.torch_device)

    def summarize(self, text):
        src_text = text
        batch = self.tokenizer.prepare_seq2seq_batch([src_text],truncation=True,padding='longest').to(self.torch_device)
        translated = self.model.generate(**batch)
        tgt_text = self.tokenizer.batch_decode(translated, skip_special_tokens=True)
        return tgt_text

summarizer = PegasusSummarizer()
print(summarizer.summarize('''(CNN)For the first time in eight years, a TV legend returned to doing what he does best. Contestants told to "come on down!" on the April 1 edition of "The Price Is Right" encountered not host Drew Carey but another familiar face in charge of the proceedings. Instead, there was Bob Barker, who hosted the TV game show for 35 years before stepping down in 2007. Looking spry at 91, Barker handled the first price-guessing game of the show, the classic "Lucky Seven," before turning hosting duties over to Carey, who finished up. Despite being away from the show for most of the past eight years, Barker didn't seem to miss a beat.'''))

```
**Output:** ['"It\'s time for the first time in a five-year anniversary of the show.']
**Output of google/pegasus-cnn_dailymail model**:['Barker hosted "The Price Is Right" for 35 years.<n>He stepped down in 2007.']


test_results.txt output:
src_pad_frac = tensor(0., device='cuda:0')
src_pad_tok = tensor(0, device='cuda:0')
step_count = 26
test_avg_gen_len = 48.63716275021758
test_avg_gen_time = 1.3503953615824382
test_avg_loss = 3.6937525272369385
test_avg_rouge1 = 19.983542428198433
test_avg_rouge2 = 4.130034786771105
test_avg_rougeL = 14.352700217580503
test_avg_rougeLsum = 18.460456248912102
test_loss = tensor(3.6938, device='cuda:0')
test_rouge2 = tensor(4.1300, device='cuda:0')
tpb = tensor(511, device='cuda:0')
val_avg_gen_len = 50.144
val_avg_gen_time = 1.513235685825348
val_avg_loss = 3.77506422996521
val_avg_rouge1 = 16.9548154
val_avg_rouge2 = 3.1666046
val_avg_rougeL = 12.980990400000001
val_avg_rougeLsum = 15.404284
val_loss = tensor(3.7751, device='cuda:0')
val_rouge2 = tensor(3.1666, device='cuda:0')


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect the output to be much cleaner and higher Rouge score. Any help in this regard would be of great help.

I am trying to retrain the model by removing  **--freeze_encoder**.
<!-- A clear and concise description of what you would expect to happen. -->

I would also try without `--fp16`<|||||>Sure @sshleifer I will try without `--fp16` and update the results here. Thanks for looking into this.<|||||>Hi @sshleifer 

I ran the below command for distillation (without --fp16 as you suggested):
`python finetune.py --learning_rate=3e-5 --do_train --do_predict --val_check_interval 0.25 --n_val 500 --num_train_epochs 6 --freeze_embeds --data_dir ./cnn_dm/ --max_target_length 142 --val_max_target_length=142 --train_batch_size=1 --eval_batch_size=1 --gradient_accumulation_steps=256 --model_name_or_path sshleifer/student_pegasus_cnn_12_2 --tokenizer_name google/pegasus-cnn_dailymail --warmup_steps 500 --output_dir distilpegasus-cnn-12-2 --gpus 1  --num_workers=0  --adafactor --freeze_encoder --task summarization  --dropout 0.1 --attention_dropout 0.1 --label_smoothing 0.1 `

However, the rouge scores are not improving even after 1 epoch
`{
    "val": [
        {
            "val_avg_loss": 940.3131713867188,
            "val_avg_rouge1": 0.0,
            "val_avg_rouge2": 0.0,
            "val_avg_rougeL": 0.0,
            "val_avg_rougeLsum": 0.0,
            "val_avg_gen_time": 1.9830520153045654,
            "val_avg_gen_len": 128.0,
            "step_count": 1
        },
        {
            "val_avg_loss": 457.8860168457031,
            "val_avg_rouge1": 0.8307167999999999,
            "val_avg_rouge2": 0.0106524,
            "val_avg_rougeL": 0.8102172,
            "val_avg_rougeLsum": 0.8177266,
            "val_avg_gen_time": 1.9989106116294861,
            "val_avg_gen_len": 128.0,
            "step_count": 2
        },
        {
            "val_avg_loss": 297.9767761230469,
            "val_avg_rouge1": 2.7392655999999995,
            "val_avg_rouge2": 0.08615479999999999,
            "val_avg_rougeL": 2.4773216,
            "val_avg_rougeLsum": 2.6349664,
            "val_avg_gen_time": 1.7901806454658509,
            "val_avg_gen_len": 93.732,
            "step_count": 3
        },
        {
            "val_avg_loss": 272.0320129394531,
            "val_avg_rouge1": 4.0338778,
            "val_avg_rouge2": 0.2913826,
            "val_avg_rougeL": 3.4839722,
            "val_avg_rougeLsum": 3.7919970000000003,
            "val_avg_gen_time": 1.4304678964614868,
            "val_avg_gen_len": 47.67,
            "step_count": 4
        },
        {
            "val_avg_loss": 259.57611083984375,
            "val_avg_rouge1": 7.9237036000000005,
            "val_avg_rouge2": 0.7740864000000001,
            "val_avg_rougeL": 6.5176862,
            "val_avg_rougeLsum": 7.265688,
            "val_avg_gen_time": 1.3813148093223573,
            "val_avg_gen_len": 37.046,
            "step_count": 5
        }
    ]
}
`

After 1 epoch, rogue2 score is 0.77. Could you please help if I am doing something wrong here?

Thanks in advance for your help.

Regards,
Karthik

<|||||>+ Note that scores are improving, just very slowly.
+ I have not had good luck with `sshleifer/student_pegasus_cnn_12_2`, I'd try to make your own student with a full encoder and a 4+ layer decoder starting. Using, for example:

```bash
python make_student.py sshleifer/pegasus-cnn-ft-v2 -save_path student_peg_cnn_16_4 -e 16 -d 4
```

Here is the [wandb log](https://wandb.ai/sshleifer/pegasus_ft/runs/32ov7btf?workspace=user-)  for a run that used `student_peg_cnn_16_4` 

![image](https://user-images.githubusercontent.com/6045025/96630846-957ff800-12e3-11eb-9956-40d911cfcfd9.png)


I started at `--max_target_length 56` and then finetuned more with `--max_target_length 142`. That log is the first run. The second run is [here](https://wandb.ai/sshleifer/pegasus_ft/runs/2z1t4r0t?workspace=user-)


FWIW, XSUM trains much faster!<|||||>Thanks @sshleifer for your inputs.

I am using this model (https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4) which has 16 encoders and 4 decoders. I am trying to reduce the inference runtime of the model - for this reason, I am trying distillation with lesser encoders and decoders.

Could you please suggest if I should try something different to reduce the inference runtime? 

Regards,
Karthik
<|||||>Try generating with the 16/4 model and `num_beams=2`.
<|||||>Thanks @sshleifer for your suggestion. This improved the runtime. Please let me know if you have more such ideas.
<|||||>Besides that, all that's easy is to make your input documents shorter, or make your generations shorter (with min_length, max_length).
