<p dir="auto">Dear <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/stas00/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stas00">@stas00</a> and whoever else is willing to help!</p>
<p dir="auto">So far I have only checked pegasus' rouge scores on 2/12 datasets for which we have checkpoints.<br>
For the other 10 datasets I either haven't tried or have tried briefly and gotten stuck.</p>
<p dir="auto">The full scope of the project is that:<br>
for each dataset:</p>
<ol dir="auto">
<li>There is an automated way to download the data, either from S3 or source. (To the extent possible, much of the logic in this script should eventually live in the <code class="notranslate">datasets</code> package).</li>
<li>we know our pegasus implementation's rouge score<br>
2b) if our score is very different than the authors', we know whether that difference is due to data preprocessing, and if it is, we can preprocess the dataset similarly to the pegasus authors.</li>
<li>Our rouge score is within 0.3 Rouge2 of the reported. (Authors) column below.</li>
</ol>
<h3 dir="auto">Steps</h3>
<h4 dir="auto">Getting Data</h4>
<p dir="auto">By far the most difficult part of each project is getting the dataset. And giving up quickly if you can't and writing a github issue somewhere.<br>
I tried 1 approach to getting data: <a href="https://gist.github.com/sshleifer/c4aed7bf4418b50caee731e94be05d9f">this script</a><br>
It worked for gigaword, I just haven't done the evaluation, but it failed for <code class="notranslate">aeslc</code> and then I gave up.</p>
<p dir="auto">Another complementary approach would be to try to directly use the <a href="https://github.com/google-research/pegasus/blob/master/pegasus/data/public_supervised_datasets.py">pegasus dataset code</a></p>
<p dir="auto">This will likely push preprocessing issues towards the back of the project. (when we try to send PRs to the datasets repo), but might be better than using my script.</p>
<h4 dir="auto">After you get data</h4>
<p dir="auto">When you have gotten a dataset you can sanity check</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="python -m torch.distributed.launch --nproc_per_node=2  run_distributed_eval.py \
	--model_name google/pegasus-large $\  # see note 1
    --save_dir xsum_generations \
    --data_dir xsum \
	--prefix test \
	--n_obs 100 \"><pre class="notranslate">python -m torch.distributed.launch --nproc_per_node=2  run_distributed_eval.py \
	--model_name google/pegasus-large $<span class="pl-cce">\ </span> <span class="pl-c"><span class="pl-c">#</span> see note 1</span>
    --save_dir xsum_generations \
    --data_dir xsum \
	--prefix <span class="pl-c1">test</span> \
	--n_obs 100 \</pre></div>
<p dir="auto">Note 1: you can just keep running pegasus-large and expect a high single digits or better rouge2 score,to avoid downloading all the checkpoints or, you can change this to the relevant checkpoint.<br>
Note 2: I am happy to run all the evals on newer hardware, very easy for me.<br>
Note 3: We can do data sharing by getting you aws creds, or some other solution. Key is that I can download from command line, e.g. Google Drive + gdown.</p>
<h3 dir="auto">Misc thoughts:</h3>
<ul dir="auto">
<li>arxiv and pubmed are listed under <code class="notranslate">scientific_papers</code> in the datasets package.</li>
<li>This is really 10 projects (1 each dataset, 2 of which I've started). If I were you I would ignore the started 2 and start on a few other ones.</li>
<li>If a dataset only has train/test or train/val or some other splits, see how the pegasus authors did the split.</li>
<li>Partial credit is valuable!</li>
<li>this could easily have been an issue for the datasets project rather than the transformers project.</li>
<li>There is no reason to merge PRs quickly for this project, but eventually we want a (much better) download_summ_dataset.py script or instructions for using other libs to accomplish the same outcome.</li>
<li>Will be good for both of us to learn the datasets internals.</li>
<li>Raw Billsum has multiple line articles, which breaks everything :( , (we could try to support raw nlp datasets in our <code class="notranslate">DataLoader</code>)</li>
</ul>
<p dir="auto">Here is a copy of the table we are trying to fill out in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="689259666" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/6844" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/6844/hovercard" href="https://github.com/huggingface/transformers/issues/6844">#6844</a> : (I made a new issue to avoid spamming that one)</p>
<table role="table">
<thead>
<tr>
<th>dataset</th>
<th>Authors</th>
<th>This Repo</th>
</tr>
</thead>
<tbody>
<tr>
<td>xsum</td>
<td>47.60/24.83/39.64</td>
<td>46.87/24.46/39.15</td>
</tr>
<tr>
<td>cnn_dailymail</td>
<td>44.16/21.56/41.30</td>
<td>see 1</td>
</tr>
<tr>
<td>newsroom</td>
<td>45.07/33.39/41.28</td>
<td>have <code class="notranslate">.tar</code> file</td>
</tr>
<tr>
<td>multi_news</td>
<td>47.65/18.75/24.95</td>
<td></td>
</tr>
<tr>
<td>gigaword</td>
<td>39.65/20.47/36.76</td>
<td>39.79/20.56/36.80</td>
</tr>
<tr>
<td>wikihow</td>
<td>46.39/22.12/38.41 *</td>
<td>Asked Authors</td>
</tr>
<tr>
<td>reddit_tifu</td>
<td>27.99/9.81/22.94</td>
<td>32.75/11.68/24.97</td>
</tr>
<tr>
<td>big_patent</td>
<td>52.29/33.08/41.66 *</td>
<td></td>
</tr>
<tr>
<td>arxiv</td>
<td>44.21/16.95/25.67</td>
<td></td>
</tr>
<tr>
<td>pubmed</td>
<td>45.97/20.15/28.25</td>
<td></td>
</tr>
<tr>
<td>aeslc</td>
<td>37.68/21.25/36.51</td>
<td>37.1/21.4/35.94</td>
</tr>
<tr>
<td>billsum</td>
<td>59.67/41.58/47.59</td>
<td>54.99/37.43/43.07</td>
</tr>
</tbody>
</table>
<p dir="auto">Originally from mixed &amp; stochastic column of this <a href="https://github.com/google-research/pegasus#results-update">table</a></p>
<p dir="auto">This was really long, and probably disorganized, so feel free to ask clarifying questions here or on slack!<br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/stas00/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stas00">@stas00</a></p>
<ol dir="auto">
<li>I got similar scores on cnn-dailymail by finetuning the authors' model on our dataset for a bit.</li>
<li>reddit_tifu: added <code class="notranslate">--min_length 32</code></li>
</ol>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.17.0</li>
<li>Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9</li>
<li>Python version: 3.6.13</li>
<li>PyTorch version (GPU?): 1.7.1 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: Using accelerate but there's only 1 GPU currently</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/SaulLu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/SaulLu">@SaulLu</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a></p>
<p dir="auto">--&gt;</p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): YituTech/conv-bert-base</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below) . Modified scripts but save_model method which is causing issues is exact same as this: <a href="https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460">https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460</a></li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below) - Similar GLUE like paired sentence classification task</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">For some models such as the one mentioned above, I've seen some times the script abruptly fails with a weird PanicException while trying to save the tokenizer. I don't think this is model specific issue, because other 3p models have saved successfully for me always, so looks like something to do with Transformers/tokenizers lib. See code that I'm using to save the model below, along with detailed logs as well.</p>
<p dir="auto">I've no idea about this issue, and how to fix it. For other standard models like bert, roberta etc the saving has worked fine always, along with some other 3p models that I've tried as well. Please let me know if you any need more information from me on this?</p>
<p dir="auto">I'm saving both the model and tokenizer only in main_process, I hope that's right? I find this slightly weird because clearly if you see from the logs below model is getting saved to dir successfully, but tokenizer specifically is failing. Not sure what could be the peculiarity here causing this? Any suggestions that you might have to fix this would be really helpful!</p>
<p dir="auto">Relevant code to save model, which is causing the issue - same as this: <a href="https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460">https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460</a> . I've also included a couple of lines at the top in the code blurb below to show how I'm initializing the config/tokenizer/model if that might be of any relevance:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    config = AutoConfig.from_pretrained(args.model_name_or_path, num_labels=num_labels, finetuning_task=None)    
    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, use_fast=not args.use_slow_tokenizer)
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(&quot;.ckpt&quot; in args.model_name_or_path),
        config=config,
        cache_dir=args.output_dir
    )

    def save_model(model, tokenizer, args):
        if args.output_dir is not None and accelerator.is_main_process:
            # Handle the repository creation
            if args.output_dir is not None:
                os.makedirs(args.output_dir, exist_ok=True)
            accelerator.wait_for_everyone()
            unwrapped_model = accelerator.unwrap_model(model)
            unwrapped_model.save_pretrained(args.output_dir, save_function=accelerator.save)
            tokenizer.save_pretrained(args.output_dir)"><pre class="notranslate"><code class="notranslate">    config = AutoConfig.from_pretrained(args.model_name_or_path, num_labels=num_labels, finetuning_task=None)    
    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, use_fast=not args.use_slow_tokenizer)
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
        cache_dir=args.output_dir
    )

    def save_model(model, tokenizer, args):
        if args.output_dir is not None and accelerator.is_main_process:
            # Handle the repository creation
            if args.output_dir is not None:
                os.makedirs(args.output_dir, exist_ok=True)
            accelerator.wait_for_everyone()
            unwrapped_model = accelerator.unwrap_model(model)
            unwrapped_model.save_pretrained(args.output_dir, save_function=accelerator.save)
            tokenizer.save_pretrained(args.output_dir)
</code></pre></div>
<p dir="auto">Logs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="03/18/2022 11:51:47 - INFO - __main__ - ***** Running training *****
03/18/2022 11:51:47 - INFO - __main__ -   Num examples = 25227
03/18/2022 11:51:47 - INFO - __main__ -   Num Epochs = 10
03/18/2022 11:51:47 - INFO - __main__ -   Instantaneous batch size per device = 32
03/18/2022 11:51:47 - INFO - __main__ -   Total train batch size (w. parallel, distributed &amp; accumulation) = 32
03/18/2022 11:51:47 - INFO - __main__ -   Gradient Accumulation steps = 1
03/18/2022 11:51:47 - INFO - __main__ -   Total optimization steps = 7890
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [28:11&lt;00:00,  6.46it/s]03/18/2022 12:20:32 - INFO - __main__ - *** Predict ***
Configuration saved in conv-bert-base/config.json
Model weights saved in conv-bert-base/pytorch_model.bin
tokenizer config file saved in conv-bert-base/tokenizer_config.json
Special tokens file saved in conv-bert-base/special_tokens_map.json
thread '&lt;unnamed&gt;' panicked at 'no entry found for key', /__w/tokenizers/tokenizers/tokenizers/src/models/mod.rs:36:66
Traceback (most recent call last):
  File &quot;run.py&quot;, line 566, in &lt;module&gt;
    main()
  File &quot;run.py&quot;, line 563, in main
    tokenizer.save_pretrained(args.output_dir)
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py&quot;, line 2108, in save_pretrained
    filename_prefix=filename_prefix,
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py&quot;, line 599, in _save_pretrained
    self.backend_tokenizer.save(tokenizer_file)
pyo3_runtime.PanicException: no entry found for key
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [29:26&lt;00:00,  4.47it/s]
Traceback (most recent call last):
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/accelerate&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/accelerate_cli.py&quot;, line 41, in main
    args.func(args)
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py&quot;, line 384, in launch_command
    simple_launcher(args)
  File &quot;/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py&quot;, line 142, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/python', 'run.py', '--model_name_or_path', 'YituTech/conv-bert-base', '--train_file', '../data/train.csv', '--validation_file', '../data/val.csv', '--test_file', '../data/test.csv', '--max_length', '128', '--per_device_train_batch_size', '32', '--learning_rate', '2e-5', '--num_train_epochs', '10', '--output_dir', 'conv-bert-base/']' returned non-zero exit status 1."><pre class="notranslate"><code class="notranslate">03/18/2022 11:51:47 - INFO - __main__ - ***** Running training *****
03/18/2022 11:51:47 - INFO - __main__ -   Num examples = 25227
03/18/2022 11:51:47 - INFO - __main__ -   Num Epochs = 10
03/18/2022 11:51:47 - INFO - __main__ -   Instantaneous batch size per device = 32
03/18/2022 11:51:47 - INFO - __main__ -   Total train batch size (w. parallel, distributed &amp; accumulation) = 32
03/18/2022 11:51:47 - INFO - __main__ -   Gradient Accumulation steps = 1
03/18/2022 11:51:47 - INFO - __main__ -   Total optimization steps = 7890
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [28:11&lt;00:00,  6.46it/s]03/18/2022 12:20:32 - INFO - __main__ - *** Predict ***
Configuration saved in conv-bert-base/config.json
Model weights saved in conv-bert-base/pytorch_model.bin
tokenizer config file saved in conv-bert-base/tokenizer_config.json
Special tokens file saved in conv-bert-base/special_tokens_map.json
thread '&lt;unnamed&gt;' panicked at 'no entry found for key', /__w/tokenizers/tokenizers/tokenizers/src/models/mod.rs:36:66
Traceback (most recent call last):
  File "run.py", line 566, in &lt;module&gt;
    main()
  File "run.py", line 563, in main
    tokenizer.save_pretrained(args.output_dir)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 2108, in save_pretrained
    filename_prefix=filename_prefix,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 599, in _save_pretrained
    self.backend_tokenizer.save(tokenizer_file)
pyo3_runtime.PanicException: no entry found for key
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [29:26&lt;00:00,  4.47it/s]
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/accelerate", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/accelerate_cli.py", line 41, in main
    args.func(args)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 384, in launch_command
    simple_launcher(args)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 142, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/python', 'run.py', '--model_name_or_path', 'YituTech/conv-bert-base', '--train_file', '../data/train.csv', '--validation_file', '../data/val.csv', '--test_file', '../data/test.csv', '--max_length', '128', '--per_device_train_batch_size', '32', '--learning_rate', '2e-5', '--num_train_epochs', '10', '--output_dir', 'conv-bert-base/']' returned non-zero exit status 1.
</code></pre></div>

<h2 dir="auto">Expected behavior</h2>

<p dir="auto">Tokenizer save should be successful.</p>