<p dir="auto">As we were planning to add <code class="notranslate">--max_train_samples  --max_val_samples  --max_test_samples</code>  to all examples <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="817565907" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/10423" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/10423/hovercard" href="https://github.com/huggingface/transformers/issues/10423">#10423</a>, I thought is there any reason why we don't expand the Trainer to handle that?</p>
<p dir="auto">It surely would be useful to be able to truncate the dataset at the point of Trainer to enable quick testing.</p>
<p dir="auto">Another plus is that the metrics can then automatically include the actual number of samples run, rather than how it is done at the moment in examples.</p>
<p dir="auto">That way this functionality would be built-in and examples will get it for free.</p>
<p dir="auto">TODO:</p>
<ol class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> port  <code class="notranslate">--max_train_samples  --max_val_samples  --max_test_samples</code>  to Trainer and remove the then unneeded code in <code class="notranslate">run_seq2seq.py</code></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> extend metrics to report  the number of samples as it's done now in:</li>
</ol>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/aca6288ff42cebded5421020f0ff088adeb446dd/examples/seq2seq/run_seq2seq.py#L590">transformers/examples/seq2seq/run_seq2seq.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 590
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/aca6288ff42cebded5421020f0ff088adeb446dd">aca6288</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L590" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="590"></td>
          <td id="LC590" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">metrics</span>[<span class="pl-s">"train_samples"</span>] <span class="pl-c1">=</span> <span class="pl-en">min</span>(<span class="pl-s1">max_train_samples</span>, <span class="pl-en">len</span>(<span class="pl-s1">train_dataset</span>)) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">so that all scripts automatically get this metric reported. Most likely it should be done here:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/aca6288ff42cebded5421020f0ff088adeb446dd/src/transformers/trainer_utils.py#L224">transformers/src/transformers/trainer_utils.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 224
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/aca6288ff42cebded5421020f0ff088adeb446dd">aca6288</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L224" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="224"></td>
          <td id="LC224" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">speed_metrics</span>(<span class="pl-s1">split</span>, <span class="pl-s1">start_time</span>, <span class="pl-s1">num_samples</span><span class="pl-c1">=</span><span class="pl-c1">None</span>): </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a></p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.6.1</li>
<li>Platform: Linux Mint Tricia 19.3 (ubuntu 18.04)</li>
<li>Python version: 3.8.8</li>
<li>PyTorch version (GPU?): 1.7.0, gpu yes</li>
<li>Tensorflow version (GPU?):</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto">tokenizer: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): GPT2</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below) text generation</li>
</ul>
<p dir="auto">After upgrade to 4.6.1 (same error in 4.6.0), I have an error when I load tokenizer.</p>
<h3 dir="auto">What I have tried</h3>
<p dir="auto">I searched for a similar issue and thought that this is a possible duplicate of <a href="https://github.com/huggingface/transformers/issues/11713" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/11713/hovercard">this issue</a>, but there was no change after I apply the solution.</p>
<p dir="auto">I uninstalled transformers and tokenizers package, reinstall those, and still there is the same issue.</p>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Import tokenizer (like below)</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import (PreTrainedTokenizerFast,
                          GPT2Tokenizer,)
"><pre class="notranslate"><code class="notranslate">from transformers import (PreTrainedTokenizerFast,
                          GPT2Tokenizer,)

</code></pre></div>
<p dir="auto">Error message</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
&lt;ipython-input-5-dc540cd053e1&gt; in &lt;module&gt;
----&gt; 1 from transformers import (PreTrainedTokenizerFast,
      2                           PreTrainedTokenizer,
      3                           AutoTokenizer,
      4                           GPT2Tokenizer,)
      5 

/opt/conda/lib/python3.8/site-packages/transformers/__init__.py in &lt;module&gt;
     41 
     42 # Check the dependencies satisfy the minimal versions required.
---&gt; 43 from . import dependency_versions_check
     44 from .file_utils import (
     45     _BaseLazyModule,

/opt/conda/lib/python3.8/site-packages/transformers/dependency_versions_check.py in &lt;module&gt;
     39                 continue  # not required, check version only if installed
     40 
---&gt; 41         require_version_core(deps[pkg])
     42     else:
     43         raise ValueError(f&quot;can't find {pkg} in {deps.keys()}, check dependency_versions_table.py&quot;)

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in require_version_core(requirement)
    118     &quot;&quot;&quot;require_version wrapper which emits a core-specific hint on failure&quot;&quot;&quot;
    119     hint = &quot;Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master&quot;
--&gt; 120     return require_version(requirement, hint)
    121 
    122 

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in require_version(requirement, hint)
    112     if want_ver is not None:
    113         for op, want_ver in wanted.items():
--&gt; 114             _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
    115 
    116 

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
     47         raise ValueError(&quot;want_ver is None&quot;)
     48     if not ops[op](version.parse(got_ver), version.parse(want_ver)):
---&gt; 49         raise ImportError(
     50             f&quot;{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}&quot;
     51         )

ImportError: tokenizers&gt;=0.10.1,&lt;0.11 is required for a normal functioning of this module, but found tokenizers==0.8.1rc1.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
&lt;ipython-input-5-dc540cd053e1&gt; in &lt;module&gt;
----&gt; 1 from transformers import (PreTrainedTokenizerFast,
      2                           PreTrainedTokenizer,
      3                           AutoTokenizer,
      4                           GPT2Tokenizer,)
      5 

/opt/conda/lib/python3.8/site-packages/transformers/__init__.py in &lt;module&gt;
     41 
     42 # Check the dependencies satisfy the minimal versions required.
---&gt; 43 from . import dependency_versions_check
     44 from .file_utils import (
     45     _BaseLazyModule,

/opt/conda/lib/python3.8/site-packages/transformers/dependency_versions_check.py in &lt;module&gt;
     39                 continue  # not required, check version only if installed
     40 
---&gt; 41         require_version_core(deps[pkg])
     42     else:
     43         raise ValueError(f"can't find {pkg} in {deps.keys()}, check dependency_versions_table.py")

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in require_version_core(requirement)
    118     """require_version wrapper which emits a core-specific hint on failure"""
    119     hint = "Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master"
--&gt; 120     return require_version(requirement, hint)
    121 
    122 

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in require_version(requirement, hint)
    112     if want_ver is not None:
    113         for op, want_ver in wanted.items():
--&gt; 114             _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
    115 
    116 

/opt/conda/lib/python3.8/site-packages/transformers/utils/versions.py in _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
     47         raise ValueError("want_ver is None")
     48     if not ops[op](version.parse(got_ver), version.parse(want_ver)):
---&gt; 49         raise ImportError(
     50             f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}"
     51         )

ImportError: tokenizers&gt;=0.10.1,&lt;0.11 is required for a normal functioning of this module, but found tokenizers==0.8.1rc1.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Just work like before!</p>