Pegasus finetuning diary
Best score so far .2065 Rouge, much worse than paper. Generations appear to start lower case/be missing words at the beginning.



Clues:
- adding `<pad>` as prefix (like for generation) makes loss nan for at least 1000 steps (I killed it). 
- Without prefix, loss is nan for 5 steps, them improves.
- distillation with teacher produces huge hidden state MSE losses. This is probably unrelated and caused by the same large activations that break fp16.

Suspects:
- different causal mask than tf?
- tf doesn't shift labels or add a decoder prefix token. We shift labels but don't add a prefix token. there is a suraj issue where this appears to be suboptimal for t5 (which also has no bos token).
- bug in label smoothing


Best run:

![image](https://user-images.githubusercontent.com/6045025/89748679-5c7d0900-da92-11ea-9505-59ee413193d9.png)

![image](https://user-images.githubusercontent.com/6045025/89748681-5dae3600-da92-11ea-94bd-5043c586bd9c.png)

freeze_embeds didn't matter (before mask fix)
mask fix explanation: 
we need `decoder_start_token_id=pad_token_id` to avoid the first word issue, but `decoder_padding_mask` should NOT tell the model to ignore that decoder_start_token_id, else you get nan.

This fix makes 1 example per batch not have eos in decoder_input_ids (t5,bart=same problem).
But maybe that can explain batch_size=1 truncation.

Original repo uses adafactor.<|||||>pegasus finetuning Running on fork branch has rouge2 23 with full beam search after 1.5 epochs
https://app.wandb.ai/sshleifer/transformers_fork-examples_seq2seq/runs/3cz2fe87?workspace=user-sshleifer

XSUM Metrics from today:
Models train on hack-pegasus-batches branch.
```
finetune: {'rouge1': 45.6515, 'rouge2': 22.9858, 'rougeL': 37.7569, 'n_obs': 11333, 'runtime': 4175.217807531357, 'seconds_per_sample': 0.3684}
dpx8 {'rouge1': 45.9739, 'rouge2': 23.1417, 'rougeL': 38.1625, 'n_obs': 11333, 'runtime': 2207.9071719646454, 'seconds_per_sample': 0.1948}
dpx4 {'rouge1': 43.0961, 'rouge2': 20.1954, 'rougeL': 35.5679, 'n_obs': 11333, 'runtime': 1813.8934507369995, 'seconds_per_sample': 0.1601}	
```

(10% chance 1st two rows are flipped)<|||||>Adafactor saves a lot of memory. All of those runs use adafactor.