[examples] Verify marian and mbart BLEU scores with examples/seq2seq/run_eval.py
Got MBART 26.8 before postprocessing on wmt_en_ro,  37.1 after.
6 minutes. (first # same as fairseq, second should be also).
Marian: 27.7/37.4 90 Seconds
Surprised. Maybe the opus data overlaps the wmt test set?<|||||>![image](https://user-images.githubusercontent.com/6045025/90946657-faf75b80-e3fc-11ea-856a-dc598d992112.png)
<|||||>Opus #s are legit!<|||||>@sshleifer do you using the default configuration in Readme ? I didn't get 37.1, and I have tried the configuration mentioned as original paper, I have got a 36.6 after postprocessing wmt_en_ro enro.

In addition, I have a confusion about the code. If I don't add extra parameters, there is a warning in the output log. I try to solve it. I can't set a breakpoint at the place where the warning is generated. It's difficult for me to locate the reason. Can you help me look at it?

`Keyword arguments {'add_prefix_space': False} not recognized.
`<|||||>(1) Which split, val or test?
(2) What was the score before post-processing?

The warning is from [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L261), you can safely ignore it.
<|||||>### Results on wmt_en_ro/test
```bash
python run_eval.py Helsinki-NLP/opus-mt-en-ro wmt_en_ro/test.source enro_test_translations.txt --reference_path wmt_en_ro/test.target -
-task translation --score_path mar_test_bleu.json --fp16 --bs 64
{'bleu': 27.6865, 'n_obs': 1999, 'runtime': 85, 'seconds_per_sample': 0.0425}
```
```bash
ro_post_process enro_finetune/test_generations.txt wmt_en_ro/test.target
# 37.4
```

### Postprocessing Setup
```bash
cd $HOME
git clone git@github.com:moses-smt/mosesdecoder.git
cd mosesdecoder  
git clone git@github.com:rsennrich/wmt16-scripts.git
ro_post_process () {
  sys=$1
  ref=$2
  export MOSES_PATH=$HOME/mosesdecoder
  REPLACE_UNICODE_PUNCT=$MOSES_PATH/scripts/tokenizer/replace-unicode-punctuation.perl
  NORM_PUNC=$MOSES_PATH/scripts/tokenizer/normalize-punctuation.perl
  REM_NON_PRINT_CHAR=$MOSES_PATH/scripts/tokenizer/remove-non-printing-char.perl
  REMOVE_DIACRITICS=$MOSES_PATH/wmt16-scripts/preprocess/remove-diacritics.py
  NORMALIZE_ROMANIAN=$MOSES_PATH/wmt16-scripts/preprocess/normalise-romanian.py
  TOKENIZER=$MOSES_PATH/scripts/tokenizer/tokenizer.perl



  lang=ro
  for file in $sys $ref; do
    cat $file \
    | $REPLACE_UNICODE_PUNCT \
    | $NORM_PUNC -l $lang \
    | $REM_NON_PRINT_CHAR \
    | $NORMALIZE_ROMANIAN \
    | $REMOVE_DIACRITICS \
    | $TOKENIZER -no-escape -l $lang \
    > $(basename $file).tok
  done
  # compute BLEU
  cat $(basename $sys).tok | sacrebleu -tok none -s none -b $(basename $ref).tok
}
```


```bash
ro_post_process enro_test_translations.txt wmt_en_ro/test.target
```
<|||||>@sshleifer That's warning can be sefaly ignore it, that is great, I have try to figure out the reason for occuring.

I using the test set, I have got 21.9 before post-processing. I'm using this postprocessing setup and run_eval.py, My analysis is that my model performance is not enough, maybe the fine-tuning parameters are inaccurate, or there are too many training steps during fine-tuning.  

I am using this version with following code link.
https://github.com/huggingface/transformers/blob/b9772897ec9f54c1a83263b059bfd37acda936d5/examples/seq2seq/finetune.py#L371

this has been changed in lastest version. 
https://github.com/huggingface/transformers/blob/9e89390ce1e785e72452207139a334cd3bf745ff/examples/seq2seq/finetune.py#L396

I am not a native English speaker, If my words is unclear, you can let me repeat it in time.
<|||||>I don't understand what code you ran, what you expected to happen, and what happened.

The change you highlight should not affect your results.<|||||>my last comment has two question.
the first question is my results can not be 37.7.
the second question is I set save_top_k ==-1,  I have solved it 

the follows is my runing parameters for the first question.
`bash train_mbart_cc25_enro_pap.sh --output_dir $OUTPUT_DIR --gpus 1 --sortish_sampler`

where train_mbart_cc25_enro_pap.sh as follows :

```
BS=16
MAX_LEN=128
python finetune.py \
    --learning_rate=3e-5 \
    --do_train \
    --val_check_interval=0.25 \
    --adam_eps 1e-06 \
    --num_train_epochs 4 --src_lang en_XX --tgt_lang ro_RO \
    --data_dir $ENRO_DIR \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
    --train_batch_size=$BS --eval_batch_size=$BS \
    --task translation \
    --warmup_steps 2500 \
    --freeze_embeds \
    --model_name_or_path=$MODEL_PATH \
    --label_smoothing 0.2 \
    --dropout 0.3 \
    "$@"
```<|||||>that's my transformers report bleu as follows,  21.9 is using sacrebleu before postprocessing. @sshleifer 
```{'bleu':` 26.0506, 'n_obs': 1999, 'runtime': 681, 'seconds_per_sample': 0.3407}```

and I have try to using the configuration as README with using `--fp16`,  and I get a error log as follows :
```
2020-09-15 15:58:46.000 [INFO] [Driver] RuntimeError: CUDA out of memory. Tried to allocate 490.00 MiB (GPU 0; 31.75 GiB total capacity; 29.17 GiB already allocated; 241.44 MiB free; 30.35 GiB reserved in total by PyTorch)
```

and I have see the README mentioned as follows

> This should take < 6h/epoch on a 16GB v100 and achieve test BLEU above 26 To get results in line with fairseq, you need to do some postprocessing. 
<|||||>+ 26.06 is definitely expected behavior, as the README indicates. 
+ The best I've ever scored from finetuning is 26.8 after training for 6 epochs. (Took 24h).
+ `--save_top-k=-1`, I can look into.
+ What command did you run with `sacrebleu` to get 21.9? The HF run_eval.py uses the `sacrebleu.corpus_bleu` python function.
+ I would recommend playing with the Helsinki-NLP/ models for faster MT finetuning.<|||||>> * 26.06 is definitely expected behavior, as the README indicates.
> * The best I've ever scored from finetuning is 26.8 after training for 6 epochs. (Took 24h).
> * `--save_top-k=-1`, I can look into.
> * What command did you run with `sacrebleu` to get 21.9? The HF run_eval.py uses the `sacrebleu.corpus_bleu` python function.
> * I would recommend playing with the Helsinki-NLP/ models for faster MT finetuning.

@sshleifer  Hi , Thank you for your kindness help. I have got a closer score 37.2 to 37.7, I have a curious question, why is `Helsinki-NLP/ models` faster ?

<|||||>They are way smaller: 74 million parameters (Marian) vs 610M (mBART)