Faster mBART finetuning
Goal: Get BLEU 20 in 1 epoch on wmt-en-ro.
Can't run a bs=1 without `--freeze_embeds`.
1 epoch takes 6H on 16GB GPU with fp16, with `--freeze_embeds` and `--freeze_encoder`. Max bs=4


Ideas:

- [ ] Dataset that fits as many sentences as possible into an example, to increase gpu utilization.
- [ ] Only store embeddings once
- [ ] prune embeddings: https://github.com/pytorch/fairseq/issues/2120
- [ ] `label_smoothing=0.1`
- [ ] TPU?


Fairseq finetune command:
https://github.com/pytorch/fairseq/issues/2179

Hi @sshleifer , can embed pruning affect accuracy(bleu) ? TPU option seems good as it won't result in less accuracy <|||||>https://github.com/pytorch/fairseq/issues/2120#issuecomment-647915216 suggests it costs 2 BLEU points.

One of our goals with examples is to make train loops that people can run on their hardware+data. If we can get 36 on a 16GB gpu that is super useful, just like `--freeze_encoder` and `--freeze_embeds` are useful, even if they probably hurt final performance a bit.
<|||||>Broke into smaller issues, but leaving this open in case people have other ideas!<|||||>note: fairseq wmt_en_de batch wps=14721.8 on V100 with https://github.com/pytorch/fairseq/issues/2506#issuecomment-678630596

check tpb/seconds for seq2seq/finetune.py

<|||||>I am trying to fine-tune on wmt-en-ro (facebook/mbart-large-cc25) using Colab and Kaggle 16GB gpu. On Colab, I get cuda oom and on Kaggle I get out of disk apace (20 GB limit). Is there a way I can skip checkpoints to fit the process in 20GB disk space? 
I even tried 1 bs, 8 max len, fp16 and freeze_encoder.<|||||>High level, I would use an opus-mt model instead of mbart for most tasks. They are smaller and tend to be nearly as good at translation, if not better. I have run ~50~ experiments finetuning mbart on wmt-en-ro and it was not a particularly pleasant experience.

Disk space: You can see what happens if you remove `checkpoint_callback` from 
https://github.com/huggingface/transformers/blob/9336086ab5d232cccd9512333518cf4299528882/examples/seq2seq/finetune.py#L362

and just call `model.model.save_pretrained`  `model.model.half().save_pretrained` after.




<|||||>Thanks for the advice @sshleifer - just wanted to try it out and test other language pairs. 
On Kaggle, as I just found out, the trick is to create the output_dir outside the working directory (where total disk space is just 5GB). Kaggle won't save it with kernel commit though.
<|||||>Cool! LMK if you have good results!<|||||>I got it working using a small subset of data, then went on training English -> Arabic translator (just a proof of concept). Model uploaded to [HF](https://huggingface.co/akhooli/mbart-large-cc25-en-ar).<|||||>Hi, I had written the code to do vocab pruning and because of the number of people wanting help with it I converted the code into a standalone library. I hope its okay if I link it here for other people to find. 

[Link to repo.](https://github.com/IamAdiSri/hf-trim)

Referencing issues #5896 #6132