distilbart-cnn reproduction
@sshleifer 
Unable to read `config.json` for `sshleifer/student_cnn_12_6`

[config.json](https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/student_cnn_12_6/config.json) Line 64 has a format error :
```
  "force_bos_token_to_be_generated", true
}
```

This should be:
 ```
 "force_bos_token_to_be_generated" : true
}
```
This is causing an issue in loading `sshleifer/student_cnn_12_6` using `.from_pretrained()`.

Fixed, will add a check. 
Are you running distillation experiments!? FYI that model is not trained.<|||||>> Are you running distillation experiments!? FYI that model is not trained.

Yes, I know. Reproducing the results, then planning to run a few experiments with it.

Wasn't able to use `--fp16 `, kept getting OOM errors (using 4 2080TIs).<|||||>Cool! 
re: fp16:
Are you in torch 1.6?
Try torch 1.5.1 with apex installed.

I haven't run anything successfully in torch 1.6 and am very suspicious of native amp.<|||||>> Try torch 1.5.1 with apex installed.
> 
> I haven't run anything successfully in torch 1.6 and am very suspicious of native amp.

Thanks, I will try that.
Also, did you use `run_eval.py` for the results [here](https://docs.google.com/spreadsheets/d/1EkhDMwVO02m8jCD1cG3RoFPLicpcL1GQHTQjfvDYgIM/edit#gid=0)? 
I tried using `sshleifer/distilbart-cnn-12-6` as well as one I finetuned from `sshleifer/student_cnn_12_6`, but got comparatively lower results.
<|||||>Yes I did, what were your results?<|||||>Validation -     `{'rouge1': 36.902390083382635, 'rouge2': 15.98520126771937, 'rougeL': 25.75566724592724} `
Test -` {'rouge1': 33.980893339399074, 'rouge2': 13.925809496977044, 'rougeL': 23.731267594610095} `<|||||>That's awful! Can I see your command?
<|||||>```
python run_eval.py distilbart-cnn-12-6/best_tfmr $DATA_DIR/val.source dbart_val_generations.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path distilbart-cnn-12-6/cnn_rouge.json \
    --task summarization \
    --n_obs 100 \
    --device cuda \
    --bs 32 \
```<|||||>On 100 observations that might not be so bad. 
The 21.26 Rouge 2 is from the following command (a few months ago):

```bash
python run_eval.py sshleifer/distilbart-cnn-12-6 \
cnn_dm/test.source \
dbart_cnn_12_6_test_gens.txt \
--reference_path cnn_dm/test.target \
--score_path dbart_cnn_12_6_test_rouge.json \
--task summarization --bs 32 --fp16
```
in torch 1.5.1.

Reran Today (it took an hour)
```
{'rouge1': 44.2503, 'rouge2': 21.2586, 'rougeL': 30.3729, 'n_obs': 11490, 'runtime': 3569, 'seconds_per_sample': 0.3106}
```


<|||||>I had tried with 1000 (based on the [comment](https://docs.google.com/spreadsheets/d/1EkhDMwVO02m8jCD1cG3RoFPLicpcL1GQHTQjfvDYgIM/edit#gid=0)),  had similar results. I wouldn't have expected the result to change that much, my bad. Thanks for your help!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
