<p dir="auto">I posted this report in the HuggingFace Forum at first, but <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/BramVanroy/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/BramVanroy">@BramVanroy</a> kindly told me to post the report here instead of the forum.<br>
The link to the post in the forum: <a href="https://discuss.huggingface.co/t/some-unintended-things-happen-in-seq2seqtrainer-example/2361" rel="nofollow">https://discuss.huggingface.co/t/some-unintended-things-happen-in-seq2seqtrainer-example/2361</a></p>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.0.0-rc-1
<ul dir="auto">
<li>The latest commit: commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/huggingface/transformers/commit/5ced23dc845c76d5851e534234b47a5aa9180d40/hovercard" href="https://github.com/huggingface/transformers/commit/5ced23dc845c76d5851e534234b47a5aa9180d40"><tt>5ced23d</tt></a></li>
</ul>
</li>
<li>Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.3</li>
<li>PyTorch version (GPU?): 1.7.0 (True)</li>
<li>Tensorflow version (GPU?): 2.3.1 (True)</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto">Trainer: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a><br>
examples/seq2seq: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): facebook/bart-large</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)</li>
</ul>
<p dir="auto">I used the XSum dataset following the README of <code class="notranslate">examples/seq2seq</code>.</p>
<h2 dir="auto">To reproduce</h2>
<h3 dir="auto">What seems strange</h3>
<ul dir="auto">
<li>The number of data pairs is not correctly recognized.</li>
<li>MLflow cannot treat the params (too long).</li>
</ul>
<p dir="auto">I wasn’t sure if I should divide these into two issues, but in the end, I decided on one.<br>
If it is better to divide them into two, I will modify it.</p>
<p dir="auto">I first noticed this strangeness when I use a different dataset than the those in the example.<br>
I again follow the README of <code class="notranslate">examples/seq2seq</code> to check if my modification causes the problem or not.</p>
<p dir="auto">Having checked <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="751327544" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/8792" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/8792/hovercard" href="https://github.com/huggingface/transformers/issues/8792">#8792</a>, I used <code class="notranslate">--evaluation_strategy epoch</code> instead of <code class="notranslate">--evaluate_during_training</code>.</p>
<h3 dir="auto">Run official example scripts</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
    --data_dir $XSUM_DIR \
    --learning_rate=3e-5 \
    --fp16 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy epoch \
    --predict_with_generate \
    --n_val 1000 \
    --model_name_or_path facebook/bart-large \
    --output_dir ./xsum_bart-large/ \
    --save_total_limit 5 \
    2&gt;&amp;1 | tee tmp.log"><pre class="notranslate"><code class="notranslate">$ CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
    --data_dir $XSUM_DIR \
    --learning_rate=3e-5 \
    --fp16 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy epoch \
    --predict_with_generate \
    --n_val 1000 \
    --model_name_or_path facebook/bart-large \
    --output_dir ./xsum_bart-large/ \
    --save_total_limit 5 \
    2&gt;&amp;1 | tee tmp.log
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<h3 dir="auto">Log</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[INFO|trainer.py:667] 2020-11-30 08:10:43,836 &gt;&gt; ***** Running training *****
[INFO|trainer.py:668] 2020-11-30 08:10:43,836 &gt;&gt;   Num examples = 204016
[INFO|trainer.py:669] 2020-11-30 08:10:43,836 &gt;&gt;   Num Epochs = 3
[INFO|trainer.py:670] 2020-11-30 08:10:43,836 &gt;&gt;   Instantaneous batch size per device = 8
[INFO|trainer.py:671] 2020-11-30 08:10:43,836 &gt;&gt;   Total train batch size (w. parallel, distributed &amp; accumulation) = 8
[INFO|trainer.py:672] 2020-11-30 08:10:43,836 &gt;&gt;   Gradient Accumulation steps = 1
[INFO|trainer.py:673] 2020-11-30 08:10:43,836 &gt;&gt;   Total optimization steps = 76506

...

mlflow.exceptions.MlflowException: Param value '{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_leng' had length 293, which exceeded length limit of 250"><pre class="notranslate"><code class="notranslate">[INFO|trainer.py:667] 2020-11-30 08:10:43,836 &gt;&gt; ***** Running training *****
[INFO|trainer.py:668] 2020-11-30 08:10:43,836 &gt;&gt;   Num examples = 204016
[INFO|trainer.py:669] 2020-11-30 08:10:43,836 &gt;&gt;   Num Epochs = 3
[INFO|trainer.py:670] 2020-11-30 08:10:43,836 &gt;&gt;   Instantaneous batch size per device = 8
[INFO|trainer.py:671] 2020-11-30 08:10:43,836 &gt;&gt;   Total train batch size (w. parallel, distributed &amp; accumulation) = 8
[INFO|trainer.py:672] 2020-11-30 08:10:43,836 &gt;&gt;   Gradient Accumulation steps = 1
[INFO|trainer.py:673] 2020-11-30 08:10:43,836 &gt;&gt;   Total optimization steps = 76506

...

mlflow.exceptions.MlflowException: Param value '{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_leng' had length 293, which exceeded length limit of 250
</code></pre></div>
<h3 dir="auto">(Reference) Dataset length</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ cd $XSUM_DIR/
$ wc -l *
    11333 test.source
    11333 test.target
   204017 train.source
   204017 train.target
    11327 val.source
    11327 val.target
   453354 total"><pre class="notranslate">$ <span class="pl-c1">cd</span> <span class="pl-smi">$XSUM_DIR</span>/
$ wc -l <span class="pl-k">*</span>
    11333 test.source
    11333 test.target
   204017 train.source
   204017 train.target
    11327 val.source
    11327 val.target
   453354 total</pre></div>
<h3 dir="auto">Details</h3>
<h4 dir="auto">The number of examples shown</h4>
<p dir="auto">At first, I tried to use the dataset with 40,000 pairs for training, but it was shown that <code class="notranslate">Num examples = 39999</code>.<br>
I don't know why, so I've checked the example with the XSum dataset.</p>
<p dir="auto">Checking the number of lengths, it seems the XSum train set used in the example has 204017 pairs, but it is shown <code class="notranslate">Num examples = 204016</code> as above.</p>
<p dir="auto">I thought the dataset was supposed to start with the first line, but am I mistaken? For example, is the first line treated as a header?</p>
<h4 dir="auto">MLflow can not treat params in this case</h4>
<p dir="auto">As shown above, the length of <code class="notranslate">param value</code> exceeds the limit that MLflow can handle.<br>
Do I just need to change the settings of MLflow? Or, should I add some modifications to <code class="notranslate">param value</code> to be used in MLflow?</p>
<p dir="auto">Thank you in advance.</p>
<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.26.0.dev0</li>
<li>Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.28</li>
<li>Python version: 3.9.16</li>
<li>Huggingface_hub version: 0.12.0</li>
<li>PyTorch version (GPU?): 1.13.1 (False)</li>
<li>Tensorflow version (GPU?): 2.11.0 (False)</li>
<li>Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)</li>
<li>Jax version: 0.3.6</li>
<li>JaxLib version: 0.3.5</li>
<li>Using GPU in script?: False</li>
<li>Using distributed or parallel set-up in script?: False</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">When I try to add a new model as per the tutorial <a href="https://huggingface.co/docs/transformers/add_new_model" rel="nofollow">here</a>, I get the following error with the given set of inputs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ transformers-cli add-new-model-like
What is the model you would like to duplicate? Please provide the lowercase `model_type` (e.g. roberta): roberta
What is the name (with no special casing) for your new model in the paper (e.g. RoBERTa)? NewTransformer
What identifier would you like to use for the `model_type` of this model?  [newtransformer] 
What lowercase name would you like to use for the module (folder) of this model?  [newtransformer] 
What prefix (camel-cased) would you like to use for the model classes of this model (e.g. Roberta)?  [NewTransformer] 
What prefix (upper-cased) would you like to use for the constants relative to this model?  [NEWTRANSFORMER] 
What will be the name of the config class for this model?  [NewTransformerConfig] 
Please give a checkpoint identifier (on the model Hub) for this new model (e.g. facebook/roberta-base): 
Will your new model use the same processing class as roberta (RobertaTokenizer) (yes/no)? no
What will be the name of the tokenizer class for this model?  [NewTransformerTokenizer] 
Traceback (most recent call last):
  File &quot;/home/stuli/.conda/envs/bin/transformers-cli&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/scratch/gpfs/stuli/transformers/src/transformers/commands/transformers_cli.py&quot;, line 54, in main
    service = args.func(args)
  File &quot;/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py&quot;, line 1351, in add_new_model_like_command_factory
    return AddNewModelLikeCommand(config_file=args.config_file, path_to_repo=args.path_to_repo)
  File &quot;/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py&quot;, line 1382, in __init__
    ) = get_user_input()
  File &quot;/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py&quot;, line 1583, in get_user_input
    image_processor_class=image_processor_class,
UnboundLocalError: local variable 'image_processor_class' referenced before assignment"><pre class="notranslate"><code class="notranslate">$ transformers-cli add-new-model-like
What is the model you would like to duplicate? Please provide the lowercase `model_type` (e.g. roberta): roberta
What is the name (with no special casing) for your new model in the paper (e.g. RoBERTa)? NewTransformer
What identifier would you like to use for the `model_type` of this model?  [newtransformer] 
What lowercase name would you like to use for the module (folder) of this model?  [newtransformer] 
What prefix (camel-cased) would you like to use for the model classes of this model (e.g. Roberta)?  [NewTransformer] 
What prefix (upper-cased) would you like to use for the constants relative to this model?  [NEWTRANSFORMER] 
What will be the name of the config class for this model?  [NewTransformerConfig] 
Please give a checkpoint identifier (on the model Hub) for this new model (e.g. facebook/roberta-base): 
Will your new model use the same processing class as roberta (RobertaTokenizer) (yes/no)? no
What will be the name of the tokenizer class for this model?  [NewTransformerTokenizer] 
Traceback (most recent call last):
  File "/home/stuli/.conda/envs/bin/transformers-cli", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/scratch/gpfs/stuli/transformers/src/transformers/commands/transformers_cli.py", line 54, in main
    service = args.func(args)
  File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1351, in add_new_model_like_command_factory
    return AddNewModelLikeCommand(config_file=args.config_file, path_to_repo=args.path_to_repo)
  File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1382, in __init__
    ) = get_user_input()
  File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1583, in get_user_input
    image_processor_class=image_processor_class,
UnboundLocalError: local variable 'image_processor_class' referenced before assignment
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">There should be no error with the given sequence of inputs when creating a new model.</p>