[s2s] Marian beam search slow for en-de 
Hey @sshleifer (tagging because of translation) - I'm not sure whether I am misunderstanding something or this is an actual issue so apologies, but it seems like validation/eval is significantly slower than training and is a serious bottleneck when fine-tuning translation models.

I am running on Colab with V100, and trying to finetune a MarianMT model on a data set of ~10k sentences of length of up to 300, training on 90% of the data takes about 2 minutes per epoch whereas validation/eval on the remaining 10% of the data takes about 6-7 minutes without any fancy eval flags, no beamsearch etc. This results in fine-tuning being ~5x slower if performing validation every epoch (and still significantly slower even if only performing partial validation or every other epoch etc).

I am using apex and pytorch 1.5.1 as instructed in the readme and in the issues regarding apex fp16 training and bs=16 for both train and validation, different batch sizes did not seem to help. Happy to post more info but the rest is pretty similar to the seq2seq examples.
Interesting, a few pointers:

+ Beam search happens by default -- the fancy eval flags (`--eval_beams=2 --eval_max_gen_length=128 --num_val_sanity_steps=0`) make beam search faster.
+ fp16 should work with or without apex.
+ Shorter sequences make val faster.
+ When the marian models are not well trained, they can get into infinite loops and generate forever. That's why `--eval_max_gen_length` is essential.

Still, 6-7 minutes to run beam search on 1,000 sentences is shockingly slow. Try running a marian checkpoint on your val set and seeing how long that takes. 


If you share a colab 
<|||||>Hey, sorry for not clarifying some things:

- I'm aware of the eval flags and I did set them such as eval_beams=1, didn't seem to make a difference. I also explicitly specify `--check_val_every_n_epoch=1 --limit_val_batches=1.0 --val_check_interval=1.0` (and checked PL Trainer docs) to make sure that I am doing a single pass over my validation set once every epoch (or X epochs).
- Should I disable apex then? Could it be the culprit?
- Sequence length also did not seem to make much of a difference, something like linear improvement (so from length of 300 and 7 minutes to length of 200 and 5 minutes). The original models are trained with a maximum length of 500 so model should support 300 technically.
- The length flags are all set (for source, target and for train,eval). The fine-tuned model starts off at ~10 BLEU before fine-tuning and ends up at ~45 BLEU after fine-tuning, so clearly the training is working, but the validation still takes ~7 minutes throughout the process.
- Running eval.py using both a pre-trained Marian model and my own fine-tuned version still takes 7 minutes, so the same as the validation that happens during the fine-tuning process.
- Will it help if I share a Colab which reproduces this problem? Should I share it as an .ipynb file or as a link to the actual Colab URL?<|||||>
Can you send me a run_eval.py command that I can run on my machine that you expect to be faster? This is hopefully going to be the simplest manifestation of the problem. Clearly apex not the culprit if run_eval is slow.

Marian generation is a much smaller surface area than translation finetuning. Many fewer moving parts.<|||||>Thanks for the fast response and willingness to help! I made a short notebook that demonstrates the problem [here](https://colab.research.google.com/drive/11HNlWfFjzBJXDEadswwkeEhUzoh6tWFm?usp=sharing). The only meaningful difference from the actual env I run my code on is apex, which as you said should not affect eval speed.<|||||>I am busy today and tomorrow but will circle back to this if you are still stuck.
One admittedly unlikely possibility is that the numbers at the beginning of the source sentences throw the model off.

Another is that this is expected performance/speed for 1000 examples * 300 tokens * 5 beams. For example, running marian on 2000 shorter wmt examples with max_len=128, 1 GPU takes about 3 minutes. So if the seqlen cost is superlinear, as theory suggests, 6-7 minutes might not be unreasonable. For example, on [CPU](https://huggingface.co/Helsinki-NLP/opus-mt-en-de?text=230%29+There+are+two+corridors+in+the+right+hand+side+wing+of+Malchut%2C+which+divide+from+this+wing+into+two+other+nations%2C+which+are+close+to+Israel+in+the+unification%2C+to+bring+them+into+the+corridors.+Under+the+left+wing+are+two+other+corridors%2C+which+divide+into+two+other+nations%2C+Amon+and+Moav%2C+and+they+are+all+called+%E2%80%9Cliving+soul.%E2%80%9D+Previously+it+was+said+that+there+are+several+corridors%2C+and+here+it+is+said+that+there+are+only+two+on+the+right+and+two+on+the+left.+The+thing+is+that+here+it+is+only+about+the+inclusive%2C+meaning+that+there+are+two+inclusive+corridors+on+the+right%2C+for+the+nations+that+belong+to+the+right%2C+and+there+are+two+inclusive+corridors+on+the+left%2C+for+the+nations+that+belong+to+the+left.+The+two+nations+on+the+right+include+all+the+nations+on+the+right+that+relate+to+the+two+general+corridors+on+the+right+wing%2C+but+The+Zohar+does+not+explain+which+are+they.+The+two+nations+on+the+left+include+all+the+nations+on+the+left%2C+which+are+Amon+and+Moav%2C+and+relate+to+the+two+general+corridors+on+the+left+wing.+All+of+them+are+called+%E2%80%9CLiving+souls.%E2%80%9D+All+the+souls+of+proselytes+that+come+from+all+the+nations+are+called+%E2%80%9Cliving+souls.%E2%80%9D+This+is+because+they+can+receive+only+from+the+Zivug+of+Gadlut+de+ZON%2C+when+ZON+are+in+the+place+of+upper+AVI.+Then+Malchut+is+called+%E2%80%9Cliving+soul%E2%80%9D+because+there+is+the+light+of+AVI+in+her%2C+which+is+the+light+of+Haya.+And+since+the+souls+of+the+proselytes+receive+from+the+wings+of+the+living+soul%2C+they+are+called+%E2%80%9Cliving+souls%2C%E2%80%9D+as+well.) your first examples takes 13 seconds.

You could try sentence splitting:

### Sentence Splitting starter code
```python
SRC = """230) There are two corridors in the right hand side wing of Malchut, which divide from this wing into two other nations, which are close to Israel in the unification, to bring them into the corridors. Under the left wing are two other corridors, which divide into two other nations, Amon and Moav, and they are all called “living soul.” Previously it was said that there are several corridors, and here it is said that there are only two on the right and two on the left. The thing is that here it is only about the inclusive, meaning that there are two inclusive corridors on the right, for the nations that belong to the right, and there are two inclusive corridors on the left, for the nations that belong to the left. The two nations on the right include all the nations on the right that relate to the two general corridors on the right wing, but The Zohar does not explain which are they. The two nations on the left include all the nations on the left, which are Amon and Moav, and relate to the two general corridors on the left wing. All of them are called “Living souls.” All the souls of proselytes that come from all the nations are called “living souls.” This is because they can receive only from the Zivug of Gadlut de ZON, when ZON are in the place of upper AVI. Then Malchut is called “living soul” because there is the light of AVI in her, which is the light of Haya. And since the souls of the proselytes receive from the wings of the living soul, they are called “living souls,” as well.
"""

model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-de')
tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
splat = SRC.split('.')
batch = tok.prepare_seq2seq_batch(splat)
batch1 = tok.prepare_seq2seq_batch([SRC])
# time these
g0 = model.generate(**batch)
g1 = model.generate(**batch1)
```




<|||||>Thanks for taking time to check it out. Wanted to add a few more things:
- It's not specific to en-de (I just put a random Marian model in the example).
- It's difficult to split longer sentences in training/validation phase, because often periods or line breaks are in different places and the chunks do not necessarily correspond, so generating such language-paired data from real text is more difficult.
- Longer sentences often contain references and topics which would be lost when breaking them down, and thus the quality of the translation would be degraded.

Regarding the experiments you've suggested:

- I tried running eval on 1k length 128 sentences as you suggested, and it took 5.5 minutes without changing the eval parameters and under 2 minutes (~3x faster) when forcing num_beams=1. 

- However when I run fine-tuning, I see the dramatic slowdown I reported before whenever validation is included in the run, with the parameter eval_beams=1 vs eval_beams=5 helping but still not entirely accounting for the issue. I have added apex to the notebook (which seems to be required to run finetuning with fp16, otherwise I get the error `You set 'use_amp=True' but do not have apex installed`), and the full details can be seen [here](https://colab.research.google.com/drive/11HNlWfFjzBJXDEadswwkeEhUzoh6tWFm?usp=sharing).
I use the same 1k dataset for both train and val with MAX_LEN=128 for the model.
Time for 1 epoch without validation: 5 seconds.
Time for 1 epoch with validation with eval_beams=1: ~100 seconds.
Time for 1 epoch with validation with eval_beams=5: ~200 seconds.

This seems to indicate that validation/eval is ~20x slower than the actual training as a baseline, which also aligns with my experience from my actual fine-tuning experiments, so I'm wondering if this is expected behavior (for example the train_mbart_enro script performs validation 4 times per epoch, which therefore must be incredibly slow?). If this is the expected performance then feel free to close this issue and I'll just try to run less validation :)<|||||>+ For summarization, I set `--n_val 100` or so if I want faster validation.
+ the wmt_enro dataset has a very different train to validation ratio: 610,319 vs 2,000. So val is not a dominant fraction of the total cost.
+ In the new `Seq2SeqTrainer` from @patil-suraj, evaluation with beam search will be optional. Hopefully this will improve your experience.<|||||>I see, thanks again!