<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): Roberta</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">I'm trying to run <code class="notranslate">run_language_modelling.py</code> with my own tokenizer</p>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="!pip install transformers --upgrade

from transformers import AutoTokenizer
my_tokenizer = AutoTokenizer.from_pretrained(&quot;distilroberta-base&quot;)
!mkdir my_tokenizer
my_tokenizer.save_pretrained(&quot;my_tokenizer&quot;)
my_tokenizer2 = AutoTokenizer.from_pretrained(&quot;./my_tokenizer&quot;)"><pre class="notranslate"><code class="notranslate">!pip install transformers --upgrade

from transformers import AutoTokenizer
my_tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")
!mkdir my_tokenizer
my_tokenizer.save_pretrained("my_tokenizer")
my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")
</code></pre></div>
<h3 dir="auto">Stack Trace:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    246                 resume_download=resume_download,
--&gt; 247                 local_files_only=local_files_only,
    248             )

4 frames
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    260         # File, but it doesn't exist.
--&gt; 261         raise EnvironmentError(&quot;file {} not found&quot;.format(url_or_filename))
    262     else:

OSError: file ./my_tokenizer/config.json not found

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
&lt;ipython-input-16-b699c4abfe8e&gt; in &lt;module&gt;()
      3 get_ipython().system('mkdir my_tokenizer')
      4 my_tokenizer.save_pretrained(&quot;my_tokenizer&quot;)
----&gt; 5 my_tokenizer2 = AutoTokenizer.from_pretrained(&quot;./my_tokenizer&quot;)

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    184         config = kwargs.pop(&quot;config&quot;, None)
    185         if not isinstance(config, PretrainedConfig):
--&gt; 186             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    187 
    188         if &quot;bert-base-japanese&quot; in pretrained_model_name_or_path:

/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    185         &quot;&quot;&quot;
    186         config_dict, _ = PretrainedConfig.get_config_dict(
--&gt; 187             pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
    188         )
    189 

/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    268                     )
    269                 )
--&gt; 270             raise EnvironmentError(msg)
    271 
    272         except json.JSONDecodeError:

OSError: Can't load './my_tokenizer'. Make sure that:

- './my_tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'

- or './my_tokenizer' is the correct path to a directory containing a 'config.json' file"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    246                 resume_download=resume_download,
--&gt; 247                 local_files_only=local_files_only,
    248             )

4 frames
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    260         # File, but it doesn't exist.
--&gt; 261         raise EnvironmentError("file {} not found".format(url_or_filename))
    262     else:

OSError: file ./my_tokenizer/config.json not found

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
&lt;ipython-input-16-b699c4abfe8e&gt; in &lt;module&gt;()
      3 get_ipython().system('mkdir my_tokenizer')
      4 my_tokenizer.save_pretrained("my_tokenizer")
----&gt; 5 my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    184         config = kwargs.pop("config", None)
    185         if not isinstance(config, PretrainedConfig):
--&gt; 186             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    187 
    188         if "bert-base-japanese" in pretrained_model_name_or_path:

/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    185         """
    186         config_dict, _ = PretrainedConfig.get_config_dict(
--&gt; 187             pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
    188         )
    189 

/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    268                     )
    269                 )
--&gt; 270             raise EnvironmentError(msg)
    271 
    272         except json.JSONDecodeError:

OSError: Can't load './my_tokenizer'. Make sure that:

- './my_tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'

- or './my_tokenizer' is the correct path to a directory containing a 'config.json' file
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">AutoTokenizer should be able to load the tokenizer from the file.</p>
<p dir="auto">Possible duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="483113043" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/1063" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/1063/hovercard" href="https://github.com/huggingface/transformers/issues/1063">#1063</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="601711222" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/3838" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/3838/hovercard" href="https://github.com/huggingface/transformers/issues/3838">#3838</a></p>

<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version:</li>
<li>Platform: Google Colab</li>
<li>Python version: 3.6.9</li>
<li>PyTorch version (GPU?): N/A</li>
<li>Tensorflow version (GPU?): N/A</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h1 dir="auto">🐛 Bug</h1>
<p dir="auto">Hi,</p>
<p dir="auto">I noticed some strange behavior with the fast tokenizers in <code class="notranslate">v2.5.0</code>, which I think is a bug:<br>
It seems <code class="notranslate">BertTokenizerFast</code> is ignoring the <code class="notranslate">pad_to_max_length</code> argument, as shown below:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from transformers import AutoTokenizer, BertTokenizer
&gt;&gt;&gt; tok_auto = AutoTokenizer.from_pretrained('bert-base-uncased')
&gt;&gt;&gt; tok_bert = BertTokenizer.from_pretrained('bert-base-uncased')
&gt;&gt;&gt; a, b = 'Sentence 1', 'Sentence 2'
&gt;&gt;&gt; 
&gt;&gt;&gt; tok_bert.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102, 0, 0, 0]  # &lt;-- Expected behavior
&gt;&gt;&gt; tok_auto.encode(a, b, max_length=10, pad_to_max_length=True)
[101, 6251, 1015, 102, 6251, 1016, 102] # &lt;-- Actual behavior"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>, <span class="pl-v">BertTokenizer</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_auto</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'bert-base-uncased'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_bert</span> <span class="pl-c1">=</span> <span class="pl-v">BertTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'bert-base-uncased'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">a</span>, <span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s">'Sentence 1'</span>, <span class="pl-s">'Sentence 2'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> 
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_bert</span>.<span class="pl-en">encode</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>, <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">pad_to_max_length</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
[<span class="pl-c1">101</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1015</span>, <span class="pl-c1">102</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1016</span>, <span class="pl-c1">102</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>]  <span class="pl-c"># &lt;-- Expected behavior</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_auto</span>.<span class="pl-en">encode</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>, <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">pad_to_max_length</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
[<span class="pl-c1">101</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1015</span>, <span class="pl-c1">102</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1016</span>, <span class="pl-c1">102</span>] <span class="pl-c"># &lt;-- Actual behavior</span></pre></div>
<p dir="auto">Also, can some please explain the reason for the warning below that's raised when I set <code class="notranslate">pad_to_max_length=False</code> (which is only there in the fast tokenizer)?</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; tok_auto.encode(a, b, max_length=10, pad_to_max_length=False)
Disabled padding because no padding token set (pad_token: [PAD], pad_token_id: 0).
To remove this error, you can add a new pad token and then resize model embedding:
	tokenizer.pad_token = '&lt;PAD&gt;'
	model.resize_token_embeddings(len(tokenizer))
[101, 6251, 1015, 102, 6251, 1016, 102]
&gt;&gt;&gt; tok_bert.encode(a, b, max_length=10, pad_to_max_length=False)
[101, 6251, 1015, 102, 6251, 1016, 102] # &lt;-- No warning"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_auto</span>.<span class="pl-en">encode</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>, <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">pad_to_max_length</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
<span class="pl-v">Disabled</span> <span class="pl-s1">padding</span> <span class="pl-s1">because</span> <span class="pl-s1">no</span> <span class="pl-s1">padding</span> <span class="pl-s1">token</span> <span class="pl-en">set</span> (<span class="pl-s1">pad_token</span>: [<span class="pl-v">PAD</span>], <span class="pl-s1">pad_token_id</span>: <span class="pl-c1">0</span>).
<span class="pl-v">To</span> <span class="pl-s1">remove</span> <span class="pl-s1">this</span> <span class="pl-s1">error</span>, <span class="pl-s1">you</span> <span class="pl-s1">can</span> <span class="pl-s1">add</span> <span class="pl-s1">a</span> <span class="pl-s1">new</span> <span class="pl-s1">pad</span> <span class="pl-s1">token</span> <span class="pl-c1">and</span> <span class="pl-s1">then</span> <span class="pl-s1">resize</span> <span class="pl-s1">model</span> <span class="pl-s1">embedding</span>:
	<span class="pl-s1">tokenizer</span>.<span class="pl-s1">pad_token</span> <span class="pl-c1">=</span> <span class="pl-s">'&lt;PAD&gt;'</span>
	<span class="pl-s1">model</span>.<span class="pl-en">resize_token_embeddings</span>(<span class="pl-en">len</span>(<span class="pl-s1">tokenizer</span>))
[<span class="pl-c1">101</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1015</span>, <span class="pl-c1">102</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1016</span>, <span class="pl-c1">102</span>]
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tok_bert</span>.<span class="pl-en">encode</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>, <span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">pad_to_max_length</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
[<span class="pl-c1">101</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1015</span>, <span class="pl-c1">102</span>, <span class="pl-c1">6251</span>, <span class="pl-c1">1016</span>, <span class="pl-c1">102</span>] <span class="pl-c"># &lt;-- No warning</span></pre></div>
<p dir="auto">Thanks!</p>